Oct 30 19:13:01.320306 ip-10-0-141-5 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 19:13:01.320317 ip-10-0-141-5 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 19:13:01.320324 ip-10-0-141-5 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 19:13:01.320544 ip-10-0-141-5 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 19:13:11.570100 ip-10-0-141-5 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 19:13:11.570121 ip-10-0-141-5 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 6cc2a45c6bf04e14a980eb98357404ad -- Oct 30 19:15:11.938301 ip-10-0-141-5 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:15:12.337542 ip-10-0-141-5 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:15:12.337542 ip-10-0-141-5 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:15:12.337542 ip-10-0-141-5 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:15:12.337542 ip-10-0-141-5 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:15:12.344310 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344214 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:12.344310 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344305 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:12.344310 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344309 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:12.344310 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344313 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:12.344310 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344316 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:12.344310 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344318 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:12.344310 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344321 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344324 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344329 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344331 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344334 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344337 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344340 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344343 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344346 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344348 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344351 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344353 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344356 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344359 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344362 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344365 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344367 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344370 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344372 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344375 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:12.344553 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344377 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344380 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344383 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344385 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344388 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344390 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344398 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344401 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344404 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344407 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344409 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344412 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344414 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344417 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344421 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344424 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344427 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344429 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344432 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344435 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:12.345056 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344437 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344440 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344443 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344445 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344448 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344451 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344453 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344456 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344459 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344471 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344476 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344481 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344484 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344487 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344490 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344493 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344495 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344498 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344501 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:12.345530 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344503 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344506 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344508 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344511 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344515 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344520 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344524 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344527 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344530 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344532 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344535 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344538 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344540 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344543 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344546 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344548 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344551 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344553 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344558 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:12.346028 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344562 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344565 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344567 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344570 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344573 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344576 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344578 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344581 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344584 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344586 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344589 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344591 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344594 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344597 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344599 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344602 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344604 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344607 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344609 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:12.346478 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344612 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344615 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.344618 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345442 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345448 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345451 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345454 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345457 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345460 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345463 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345465 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345468 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345471 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345475 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345477 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345480 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345483 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345486 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345489 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:12.346913 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345492 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345495 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345497 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345499 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345502 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345504 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345507 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345509 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345512 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345515 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345518 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345521 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345524 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345526 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345529 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345532 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345535 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345538 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345540 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345543 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:12.347383 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345546 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345548 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345551 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345553 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345556 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345558 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345561 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345570 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345573 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345576 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345579 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345582 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345584 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345587 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345590 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345592 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345595 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345599 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345602 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:12.347857 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345605 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345607 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345610 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345612 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345615 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345618 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345620 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345623 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345625 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345628 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345630 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345633 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345635 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345638 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345641 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345644 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345646 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345648 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345651 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345654 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:12.348316 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345656 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345661 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345663 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345666 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345668 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345671 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345673 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345676 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345678 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345681 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345684 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345687 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345690 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345692 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345695 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345698 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345700 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345703 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345705 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345708 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:12.348791 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345712 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345716 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345719 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345722 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345724 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345729 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345732 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345735 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345738 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345740 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.345745 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346312 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346319 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346327 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346330 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346333 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346336 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346338 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346341 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346344 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:12.349277 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346346 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346349 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346354 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346357 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346360 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346362 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346365 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346368 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346371 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346374 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346376 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346379 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346382 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346384 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346387 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346389 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346392 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346396 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346399 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:12.349777 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346402 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346404 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346407 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346410 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346412 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346415 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346417 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346421 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346423 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346426 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346429 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346431 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346434 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346436 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346439 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346443 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346446 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346448 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346451 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346453 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:12.350236 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346456 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346459 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346461 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346464 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346466 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346469 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346472 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346474 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346477 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346479 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346483 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346487 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346490 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346492 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346495 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346497 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346500 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346502 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346505 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346507 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:12.350722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346514 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346516 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346519 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346521 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346524 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346526 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346529 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346532 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346535 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346538 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346540 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346543 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346545 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346548 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346550 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346553 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346556 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346558 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346561 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346563 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:12.351237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346566 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346568 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346572 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346574 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346577 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346579 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346582 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346585 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346588 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346592 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346595 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346599 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346603 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346612 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346615 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346618 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346621 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.346624 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348245 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348259 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:15:12.351703 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348269 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348274 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348278 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348282 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348289 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348294 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348297 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348300 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348304 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348307 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348310 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348313 2577 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348316 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348319 2577 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348322 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348326 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348330 2577 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348333 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348337 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348341 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348345 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348353 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348356 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:15:12.352197 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348359 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348363 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348366 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348370 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348373 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348376 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348380 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348385 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348388 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348391 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348394 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348397 2577 flags.go:64] FLAG: --enable-server="true" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348400 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348405 2577 flags.go:64] FLAG: --event-burst="100" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348408 2577 flags.go:64] FLAG: --event-qps="50" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348411 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348415 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348417 2577 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348422 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348425 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348429 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348432 2577 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348435 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348438 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348441 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:15:12.352772 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348444 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348447 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348450 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348453 2577 flags.go:64] FLAG: --feature-gates="" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348457 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348461 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348464 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348467 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348470 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348473 2577 flags.go:64] FLAG: --help="false" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348476 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348480 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348483 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348486 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348489 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348493 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348496 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348499 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348502 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348505 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348508 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348511 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348514 2577 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348517 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:15:12.353422 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348519 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348522 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348526 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348529 2577 flags.go:64] FLAG: --lock-file="" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348532 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348535 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348538 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348544 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348547 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348550 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348553 2577 flags.go:64] FLAG: --logging-format="text" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348556 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348559 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348562 2577 flags.go:64] FLAG: --manifest-url="" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348565 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348569 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348573 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348576 2577 flags.go:64] FLAG: --max-pods="110" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348580 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348583 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348592 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348595 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348600 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348603 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348606 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:15:12.354100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348614 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348618 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348621 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348624 2577 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348627 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348630 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348632 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348635 2577 flags.go:64] FLAG: --port="10250" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348638 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348641 2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-09695772b70aab656" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348644 2577 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348647 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348650 2577 flags.go:64] FLAG: --register-node="true" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348653 2577 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348657 2577 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348660 2577 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348662 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348665 2577 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348669 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348672 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348675 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348678 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348681 2577 flags.go:64] FLAG: --runonce="false" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348684 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348689 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348692 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:15:12.354685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348695 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348697 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348702 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348706 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348710 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348713 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348716 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348719 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348721 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348724 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348728 2577 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348731 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348736 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348739 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348742 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348747 2577 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348749 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348752 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348755 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348758 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348762 2577 flags.go:64] FLAG: --v="2" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348766 2577 flags.go:64] FLAG: --version="false" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348771 2577 flags.go:64] FLAG: --vmodule="" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348775 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.348779 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:15:12.355366 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.348928 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.348932 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.348935 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.348938 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.348940 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.348945 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.348948 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.348951 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.348954 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.348956 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.348961 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.348964 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.348967 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.348970 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.348973 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349011 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349016 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349019 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349021 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:12.356411 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349024 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349027 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349030 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349033 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349035 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349038 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349040 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349043 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349046 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349048 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349051 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349053 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349056 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349058 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349062 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349064 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349067 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349070 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349075 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:12.357237 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349078 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349081 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349084 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349087 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349090 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349094 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349096 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349099 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349101 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349105 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349109 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349112 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349115 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349118 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349120 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349123 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349126 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349128 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349131 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:12.358051 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349134 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349136 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349139 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349141 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349144 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349147 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349149 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349151 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349154 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349156 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349159 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349161 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349166 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349169 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349172 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349174 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349177 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349180 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349183 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349187 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:12.358868 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349190 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349192 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349195 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349197 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349200 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349203 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349205 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349208 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349210 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349213 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349215 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349218 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349220 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349223 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349225 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349228 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349230 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349233 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349235 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349238 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:12.359753 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349241 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:12.360482 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349243 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:12.360482 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349245 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:12.360482 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349248 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:12.360482 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349252 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:12.360482 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349255 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:12.360482 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349258 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:12.360482 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349261 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:12.360482 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349264 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:12.360482 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.349293 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:15:12.360482 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349331 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:12.360482 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349336 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:12.360482 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349339 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:12.360482 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349342 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:12.360482 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349345 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:12.360482 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349347 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:12.360482 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349350 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349353 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349355 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349358 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349362 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349366 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349369 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349372 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349376 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349378 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349381 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349384 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349386 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349389 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349391 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349395 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349399 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349402 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349405 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:12.360891 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349407 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349410 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349415 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349418 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349421 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349423 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349427 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349430 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349432 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349435 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349438 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349441 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349444 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349446 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349449 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349451 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349454 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349457 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349459 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349462 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:12.361653 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349464 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349467 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349470 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349472 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349474 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349477 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349480 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349483 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349485 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349488 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349490 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349493 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349495 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349498 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349502 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349504 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349507 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349510 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349512 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349515 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:12.362331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349517 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349520 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349522 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349526 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349528 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349531 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349533 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349536 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349538 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349541 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349543 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349546 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349549 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349551 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349554 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349557 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349559 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349561 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349564 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:12.362830 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349567 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349569 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349571 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349574 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349576 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349579 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349581 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349585 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349588 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349591 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349594 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349597 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349600 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349602 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349605 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349607 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349610 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349613 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349615 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:12.363331 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349618 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349620 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.349623 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.358051 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.358074 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.358098 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.358107 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358195 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358203 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358209 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358214 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358218 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358223 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358227 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358233 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358237 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358241 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358246 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358250 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:12.363814 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358255 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358259 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358264 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358268 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358273 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358277 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358281 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358285 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358289 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358294 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358298 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358302 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358307 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358311 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358315 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358319 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358324 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358330 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358338 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:12.364309 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358345 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358350 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358355 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358359 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358364 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358368 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358372 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358376 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358381 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358385 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358389 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358394 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358398 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358403 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358407 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358411 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358415 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358419 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358424 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358428 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:12.365042 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358433 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358437 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358441 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358446 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358450 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358454 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358459 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358463 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358467 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358472 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358477 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358482 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358486 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358490 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358494 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358498 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358502 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358507 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358511 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358515 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:12.365525 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358520 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358524 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358528 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358532 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358538 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358544 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358549 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358554 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358559 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358564 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358568 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358572 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358577 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358581 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358585 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358589 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358593 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358598 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358602 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:12.366285 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358606 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:12.366722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358611 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:12.366722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358615 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:12.366722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358620 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:12.366722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358625 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:12.366722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358630 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:12.366722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358636 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:12.366722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358640 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:12.366722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358645 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:12.366722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358649 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:12.366722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358653 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:12.366722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358659 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:12.366722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358663 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:12.366722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358667 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:12.366722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358671 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:12.366722 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358676 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:12.366722 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.358729 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358968 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358994 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.358999 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359004 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359009 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359014 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359018 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359023 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359028 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359032 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359037 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359041 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359045 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359049 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359054 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359060 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359067 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359071 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:12.367137 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359076 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359081 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359085 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359091 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359095 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359100 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359105 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359109 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359113 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359118 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359122 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359126 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359130 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359135 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359139 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359143 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359148 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359152 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359157 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359161 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:12.367649 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359165 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359169 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359173 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359178 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359182 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359187 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359191 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359195 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359200 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359204 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359208 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359213 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359217 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359222 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359226 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359231 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359236 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359240 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359244 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359248 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:12.368153 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359252 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359256 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359260 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359264 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359268 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359273 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359278 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359282 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359286 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359290 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359295 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359299 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359302 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359306 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359311 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359315 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359319 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359323 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359328 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:12.368651 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359334 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359340 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359345 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359349 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359354 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359358 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359363 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359367 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359371 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359402 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359409 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359415 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359421 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359425 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359430 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359434 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359438 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359444 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359448 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:12.369127 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359452 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:12.369593 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359456 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:12.369593 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359461 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:12.369593 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359465 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:12.369593 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359469 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:12.369593 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359473 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:12.369593 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359478 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:12.369593 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359482 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:12.369593 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359486 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:12.369593 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359490 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:12.369593 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.359531 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:15:12.369593 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359586 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:12.369593 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359595 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:12.369593 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359602 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:12.369593 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359607 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:12.369593 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359612 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359617 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359621 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359625 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359630 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359634 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359638 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359643 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359647 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359652 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359656 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359660 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359664 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359668 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359673 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359678 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359682 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359686 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359691 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359695 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:12.369946 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359699 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359703 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359707 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359712 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359716 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359720 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359724 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359732 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359737 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359741 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359745 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359750 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359754 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359758 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359762 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359767 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359771 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359775 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359779 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:12.370761 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359783 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359788 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359794 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359802 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359806 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359811 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359815 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359820 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359824 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359828 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359833 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359838 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359842 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359846 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359850 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359855 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359859 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359863 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359868 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359872 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:12.371503 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359878 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359883 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359887 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359891 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359895 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359900 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359904 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359908 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359912 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359917 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359921 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359925 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359930 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359934 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359938 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359942 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359947 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359951 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359955 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:12.372220 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359959 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359964 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359968 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359972 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.359994 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360001 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360005 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360009 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360014 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360018 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360023 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360029 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360033 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360040 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360045 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360049 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360053 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360057 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360062 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360066 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:12.372679 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360070 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:12.373246 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360074 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:12.373246 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360079 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:12.373246 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:12.360084 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:12.373246 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.360175 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:15:12.373246 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.365962 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 19:15:12.373246 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.367443 2577 server.go:1014] "Starting client certificate rotation" Oct 30 19:15:12.373246 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.367537 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:15:12.373246 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.367571 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:15:12.373246 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.372496 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:15:12.378492 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.378469 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:15:12.392445 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.392426 2577 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:15:12.395304 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.395290 2577 log.go:25] "Validated CRI v1 image API" Oct 30 19:15:12.396151 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.396125 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:15:12.400183 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.400161 2577 fs.go:135] Filesystem UUIDs: map[2cbb70df-2a1d-4d46-895f-43f8b25f0b0e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 caf7e14a-2aa0-430a-8b3d-1bd06f7c283a:/dev/nvme0n1p3] Oct 30 19:15:12.400256 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.400181 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 19:15:12.406343 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.406224 2577 manager.go:217] Machine: {Timestamp:2026-10-30 19:15:12.404637623 +0000 UTC m=+0.360844766 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3115646 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2906c43bfb44990648ec3708beb21b SystemUUID:ec2906c4-3bfb-4499-0648-ec3708beb21b BootID:6cc2a45c-6bf0-4e14-a980-eb98357404ad Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:58:f0:f6:81:39 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:58:f0:f6:81:39 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:71:ee:dd:5c:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:15:12.406343 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.406342 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:15:12.406445 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.406434 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:15:12.409275 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.409249 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:15:12.409421 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.409277 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-5.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:15:12.409467 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.409423 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:15:12.409467 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.409432 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:15:12.409516 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.409473 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:15:12.409516 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.409493 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:15:12.411135 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.411122 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:15:12.415939 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.415925 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:15:12.416083 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.416073 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:15:12.416863 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.416854 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:15:12.416896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.416867 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:15:12.416896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.416881 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:15:12.416896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.416890 2577 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:15:12.416972 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.416900 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:15:12.417376 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.417360 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:15:12.419777 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.419746 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:15:12.428473 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.428452 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:15:12.429530 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.429515 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:15:12.429605 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.429547 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:15:12.429710 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.429700 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:15:12.429741 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.429715 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:15:12.429741 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.429722 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:15:12.429741 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.429727 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:15:12.429741 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.429733 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:15:12.429741 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.429739 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:15:12.430573 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.430563 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:15:12.430573 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.430574 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:15:12.430638 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.430582 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:15:12.430638 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.430588 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:15:12.430638 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.430597 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:15:12.430638 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.430605 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:15:12.431903 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.431891 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:15:12.431940 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.431904 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:15:12.435787 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.435767 2577 server.go:1263] "Started kubelet" Oct 30 19:15:12.435901 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.435850 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:15:12.435938 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.435858 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:15:12.435990 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.435938 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:15:12.436545 ip-10-0-141-5 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:15:12.437156 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.437137 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:15:12.437254 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.437176 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:15:12.442726 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.442706 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 19:15:12.444799 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.444780 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:15:12.447009 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.446991 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:15:12.447135 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.447113 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-5.us-west-1.compute.internal\" not found" Oct 30 19:15:12.447906 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.447877 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:15:12.447906 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.447897 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:15:12.448045 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.447916 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:15:12.448045 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.448030 2577 factory.go:55] Registering systemd factory Oct 30 19:15:12.448100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.448052 2577 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:15:12.448129 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.448033 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:15:12.448129 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.448111 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:15:12.448315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.448301 2577 factory.go:153] Registering CRI-O factory Oct 30 19:15:12.448387 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.448318 2577 factory.go:223] Registration of the crio container factory successfully Oct 30 19:15:12.448387 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.448373 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:15:12.448491 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.448392 2577 factory.go:103] Registering Raw factory Oct 30 19:15:12.448491 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.448405 2577 manager.go:1199] Started watching for new ooms in manager Oct 30 19:15:12.449326 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.449312 2577 manager.go:319] Starting recovery of all containers Oct 30 19:15:12.462009 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.460700 2577 manager.go:324] Recovery completed Oct 30 19:15:12.462497 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.462470 2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 19:15:12.465702 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.465685 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:12.467284 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.467252 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:12.467379 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.467301 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:12.467379 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.467327 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:12.467917 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.467902 2577 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:15:12.467917 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.467910 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:12.468067 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.467912 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:15:12.468067 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.467950 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:15:12.468642 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.468628 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:12.468701 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.468652 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:12.468701 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.468662 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:12.470220 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.470209 2577 policy_none.go:50] "Start" Oct 30 19:15:12.470267 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.470224 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:15:12.470267 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.470235 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:15:12.472236 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.472222 2577 policy_none.go:44] "Start" Oct 30 19:15:12.513590 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.513554 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:15:12.516266 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.516252 2577 manager.go:341] "Starting Device Plugin manager" Oct 30 19:15:12.516329 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.516289 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:15:12.516329 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.516301 2577 server.go:87] "Starting device plugin registration server" Oct 30 19:15:12.516623 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.516610 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:15:12.516690 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.516661 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:15:12.516800 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.516772 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:15:12.517196 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.516872 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:15:12.517196 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.516885 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:15:12.517353 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.517337 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:15:12.517408 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.517395 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-5.us-west-1.compute.internal\" not found" Oct 30 19:15:12.573794 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.573766 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-5.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 19:15:12.575768 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.573768 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-5.us-west-1.compute.internal.18e3649f684857f9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-5.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-5.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:15:12.435738617 +0000 UTC m=+0.391945758,LastTimestamp:2026-10-30 19:15:12.435738617 +0000 UTC m=+0.391945758,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-5.us-west-1.compute.internal,}" Oct 30 19:15:12.606085 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.606002 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:15:12.607236 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.607206 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:15:12.607236 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.607235 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:15:12.607389 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.607259 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:15:12.607389 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.607296 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 19:15:12.617207 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.617185 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:12.618091 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.618067 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:12.618169 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.618099 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:12.618169 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.618109 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:12.618169 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.618136 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.622402 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.622380 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-5.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 19:15:12.634326 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.634293 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 19:15:12.634470 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.634315 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-5.us-west-1.compute.internal.18e3649f6a29abb8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-5.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-5.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-5.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:15:12.467282872 +0000 UTC m=+0.423490018,LastTimestamp:2026-10-30 19:15:12.467282872 +0000 UTC m=+0.423490018,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-5.us-west-1.compute.internal,}" Oct 30 19:15:12.708043 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.708007 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal"] Oct 30 19:15:12.708200 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.708117 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:12.709162 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.709136 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:12.709272 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.709171 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:12.709272 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.709185 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:12.711703 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.711542 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:12.711703 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.711699 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.711852 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.711729 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:12.712539 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.712521 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:12.712625 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.712558 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:12.712625 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.712573 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:12.712625 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.712524 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:12.712768 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.712632 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:12.712768 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.712643 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:12.714709 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.714693 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.714785 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.714720 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:12.715523 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.715510 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:12.715594 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.715536 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:12.715594 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.715546 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:12.730334 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.730245 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-5.us-west-1.compute.internal.18e3649f6a2a0c6c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-5.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-5.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-5.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:15:12.467307628 +0000 UTC m=+0.423514774,LastTimestamp:2026-10-30 19:15:12.467307628 +0000 UTC m=+0.423514774,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-5.us-west-1.compute.internal,}" Oct 30 19:15:12.730477 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.730391 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 19:15:12.730477 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.730412 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-5.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 19:15:12.742941 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.742920 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-5.us-west-1.compute.internal\" not found" node="ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.745639 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.745621 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-5.us-west-1.compute.internal\" not found" node="ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.749821 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.749805 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d17eaea73d7bf32a8a9bb9e63451f939-config\") pod \"kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"d17eaea73d7bf32a8a9bb9e63451f939\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.749898 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.749829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56c484cac7643547da8de4853003de0e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"56c484cac7643547da8de4853003de0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.749898 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.749845 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56c484cac7643547da8de4853003de0e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"56c484cac7643547da8de4853003de0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.777042 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.777012 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:15:12.777182 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.777095 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-22rk5" Oct 30 19:15:12.789300 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.789269 2577 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-141-5.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.789482 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.789303 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 19:15:12.789556 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.789459 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-5.us-west-1.compute.internal.18e3649f6a2a6fca default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-5.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-5.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-5.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:15:12.467333066 +0000 UTC m=+0.423540211,LastTimestamp:2026-10-30 19:15:12.467333066 +0000 UTC m=+0.423540211,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-5.us-west-1.compute.internal,}" Oct 30 19:15:12.851719 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.851674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d17eaea73d7bf32a8a9bb9e63451f939-config\") pod \"kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"d17eaea73d7bf32a8a9bb9e63451f939\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.851719 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.851718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56c484cac7643547da8de4853003de0e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"56c484cac7643547da8de4853003de0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.851938 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.851740 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56c484cac7643547da8de4853003de0e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"56c484cac7643547da8de4853003de0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.851938 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.851774 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d17eaea73d7bf32a8a9bb9e63451f939-config\") pod \"kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"d17eaea73d7bf32a8a9bb9e63451f939\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.851938 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.851793 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56c484cac7643547da8de4853003de0e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"56c484cac7643547da8de4853003de0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.851938 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.851777 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56c484cac7643547da8de4853003de0e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"56c484cac7643547da8de4853003de0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.936630 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:12.936542 2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-5.us-west-1.compute.internal\" not found" node="ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:12.990217 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.990176 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:12.991226 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.991206 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:12.991327 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.991238 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:12.991327 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.991249 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:12.991327 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:12.991274 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:13.045885 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.045855 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:13.047421 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.047397 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:13.047515 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:13.047430 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-5.us-west-1.compute.internal\": node \"ip-10-0-141-5.us-west-1.compute.internal\" not found" Oct 30 19:15:13.048758 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.048742 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:13.201731 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:13.201641 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-5.us-west-1.compute.internal\" not found" Oct 30 19:15:13.302770 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:13.302726 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-5.us-west-1.compute.internal\" not found" Oct 30 19:15:13.368207 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.368165 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 19:15:13.368719 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:13.368358 2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a99e23be2ae284760adacf985437c851-69fa068035f08793.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-5.us-west-1.compute.internal.18e3649f6a29abb8\": read tcp 10.0.141.5:48726->13.184.205.62:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-5.us-west-1.compute.internal.18e3649f6a29abb8 default 5010 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-5.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-5.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-5.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:15:12 +0000 UTC,LastTimestamp:2026-10-30 19:15:12.618082662 +0000 UTC m=+0.574289805,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-5.us-west-1.compute.internal,}" Oct 30 19:15:13.402908 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:13.402870 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-5.us-west-1.compute.internal\" not found" Oct 30 19:15:13.506434 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:13.506359 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-5.us-west-1.compute.internal\" not found" Oct 30 19:15:13.608373 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:13.608317 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-5.us-west-1.compute.internal\" not found" Oct 30 19:15:13.691279 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.691246 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:13.692296 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.692272 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:13.692418 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.692311 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:13.692418 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.692324 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:13.709016 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:13.708967 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-5.us-west-1.compute.internal\" not found" Oct 30 19:15:13.782027 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.781932 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:10:13 +0000 UTC" deadline="2028-08-06 08:45:42.16388439 +0000 UTC" Oct 30 19:15:13.782027 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.781973 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15493h30m28.381915515s" Oct 30 19:15:13.791429 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.791400 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:15:13.809585 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:13.809556 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-5.us-west-1.compute.internal\" not found" Oct 30 19:15:13.843225 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.843194 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:15:13.847658 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.847638 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:13.897697 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.897667 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:15:13.912310 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.912283 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:15:13.913253 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.913239 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:13.974020 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:13.973965 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:15:14.016072 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.016041 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:15:14.204003 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:14.203948 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17eaea73d7bf32a8a9bb9e63451f939.slice/crio-809989d2cacca82b7944f52ffbc7a90dc664b50616965de57f062a5f95b8bff8 WatchSource:0}: Error finding container 809989d2cacca82b7944f52ffbc7a90dc664b50616965de57f062a5f95b8bff8: Status 404 returned error can't find the container with id 809989d2cacca82b7944f52ffbc7a90dc664b50616965de57f062a5f95b8bff8 Oct 30 19:15:14.208832 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:14.208804 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c484cac7643547da8de4853003de0e.slice/crio-ffe9267f936342b7b037e91fccd9c0e4ab92c2912899d3c0edcd8500804c87eb WatchSource:0}: Error finding container ffe9267f936342b7b037e91fccd9c0e4ab92c2912899d3c0edcd8500804c87eb: Status 404 returned error can't find the container with id ffe9267f936342b7b037e91fccd9c0e4ab92c2912899d3c0edcd8500804c87eb Oct 30 19:15:14.208930 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.208831 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:15:14.418650 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.418562 2577 apiserver.go:52] "Watching apiserver" Oct 30 19:15:14.447241 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.447205 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 19:15:14.529104 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.529071 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:15:14.529420 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.529398 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-lzl7q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w","openshift-dns/node-resolver-25zlr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal","openshift-multus/multus-kldq6","openshift-network-diagnostics/network-check-target-ql75b","kube-system/global-pull-secret-syncer-6vks7","kube-system/konnectivity-agent-lv4mc","openshift-image-registry/node-ca-s4k9z","openshift-multus/multus-additional-cni-plugins-v2hmn","openshift-multus/network-metrics-daemon-7gj92","openshift-cluster-node-tuning-operator/tuned-8ch84","openshift-ovn-kubernetes/ovnkube-node-kg9n8","kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal"] Oct 30 19:15:14.533401 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.533379 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:15:14.535706 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.535682 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.537851 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.537829 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:14.540541 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.540520 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.542922 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.542906 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.546163 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.546142 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:14.550082 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.548710 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:14.551573 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.551432 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:14.551573 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:14.551429 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vks7" podUID="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" Oct 30 19:15:14.554580 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.554557 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:14.554678 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:14.554655 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ql75b" podUID="3d1a766f-5256-4044-98d5-f551f811e684" Oct 30 19:15:14.556949 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.556927 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.558531 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558515 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-var-lib-kubelet\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.558626 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558538 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.558626 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558556 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/373b617b-0896-420c-b2fd-38635c62a21c-host\") pod \"node-ca-s4k9z\" (UID: \"373b617b-0896-420c-b2fd-38635c62a21c\") " pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:14.558626 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-hosts-file\") pod \"node-resolver-25zlr\" (UID: \"ddd92762-f7c4-4f85-80e1-0fe7045e78e8\") " pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:14.558626 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558605 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a62fb314-b483-4dcd-8170-a818757da247-ovn-node-metrics-cert\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.558816 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558633 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdrqv\" (UniqueName: \"kubernetes.io/projected/bd91ed10-7ecf-415f-9906-3a79c23c8229-kube-api-access-wdrqv\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.558816 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558657 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-socket-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.558816 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558683 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-etc-selinux\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.558816 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558703 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-log-socket\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.558816 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-run\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.558816 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-tuned\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.558816 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558791 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd91ed10-7ecf-415f-9906-3a79c23c8229-tmp\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.559168 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558840 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-dbus\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:14.559168 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558872 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/373b617b-0896-420c-b2fd-38635c62a21c-serviceca\") pod \"node-ca-s4k9z\" (UID: \"373b617b-0896-420c-b2fd-38635c62a21c\") " pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:14.559168 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-var-lib-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.559168 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558917 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-systemd\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.559168 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558932 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-kubelet-config\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:14.559168 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558948 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.559168 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558963 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-kubelet\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.559168 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.558976 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-slash\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.559168 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559014 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-ovnkube-config\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.559168 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559045 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-host\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.559168 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559066 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:14.559168 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559082 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-node-log\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.559168 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-sys-fs\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.559168 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559142 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-tmp-dir\") pod \"node-resolver-25zlr\" (UID: \"ddd92762-f7c4-4f85-80e1-0fe7045e78e8\") " pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559176 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-run-netns\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-systemd\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559261 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-etc-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r5xm\" (UniqueName: \"kubernetes.io/projected/a62fb314-b483-4dcd-8170-a818757da247-kube-api-access-5r5xm\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559313 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v974s\" (UniqueName: \"kubernetes.io/projected/373b617b-0896-420c-b2fd-38635c62a21c-kube-api-access-v974s\") pod \"node-ca-s4k9z\" (UID: \"373b617b-0896-420c-b2fd-38635c62a21c\") " pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559337 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-device-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559362 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6245n\" (UniqueName: \"kubernetes.io/projected/42b67c56-566b-4319-93ff-165d6cefdf19-kube-api-access-6245n\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559386 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnzd6\" (UniqueName: \"kubernetes.io/projected/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-kube-api-access-qnzd6\") pod \"node-resolver-25zlr\" (UID: \"ddd92762-f7c4-4f85-80e1-0fe7045e78e8\") " pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559421 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-systemd-units\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559446 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-env-overrides\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559469 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-modprobe-d\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559471 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559505 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysconfig\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559526 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad299622-53d4-4610-b03a-3c586c0fff56-konnectivity-ca\") pod \"konnectivity-agent-lv4mc\" (UID: \"ad299622-53d4-4610-b03a-3c586c0fff56\") " pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559542 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-ovn\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559556 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-cni-netd\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.559663 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:14.559549 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7gj92" podUID="a619b2e9-c604-4179-aa69-aa822f209f0c" Oct 30 19:15:14.560327 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559597 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.560327 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-ovnkube-script-lib\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.560327 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559659 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysctl-d\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.560327 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559689 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-lib-modules\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.560327 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559714 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad299622-53d4-4610-b03a-3c586c0fff56-agent-certs\") pod \"konnectivity-agent-lv4mc\" (UID: \"ad299622-53d4-4610-b03a-3c586c0fff56\") " pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:15:14.560327 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559738 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-registration-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.560327 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.560327 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559805 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-cni-bin\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.560327 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559854 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-kubernetes\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.560327 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559897 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysctl-conf\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.560327 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.559929 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-sys\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.561856 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.561840 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.589595 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.589567 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:15:14.589735 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.589716 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:15:14.589811 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.589774 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q7svh\"" Oct 30 19:15:14.590749 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.590728 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:15:14.591402 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.591384 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:15:14.591492 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.591426 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:15:14.591492 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.591462 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-csh58\"" Oct 30 19:15:14.593442 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.593425 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:15:14.593646 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.593630 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:15:14.593770 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.593759 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xzg6c\"" Oct 30 19:15:14.595849 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.595835 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:15:14.596252 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.596238 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n57d7\"" Oct 30 19:15:14.596479 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.596459 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:15:14.611942 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.611885 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" event={"ID":"56c484cac7643547da8de4853003de0e","Type":"ContainerStarted","Data":"ffe9267f936342b7b037e91fccd9c0e4ab92c2912899d3c0edcd8500804c87eb"} Oct 30 19:15:14.612908 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.612884 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" event={"ID":"d17eaea73d7bf32a8a9bb9e63451f939","Type":"ContainerStarted","Data":"809989d2cacca82b7944f52ffbc7a90dc664b50616965de57f062a5f95b8bff8"} Oct 30 19:15:14.631140 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.631103 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:15:14.631301 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.631267 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:15:14.631361 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.631296 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:15:14.631361 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.631335 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:15:14.640768 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.640744 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:15:14.643311 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.643295 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rhxwm\"" Oct 30 19:15:14.643389 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.643331 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:15:14.643481 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.643450 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:15:14.643481 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.643472 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:15:14.643636 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.643491 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:15:14.643636 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.643529 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5vtr8\"" Oct 30 19:15:14.643995 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.643981 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6brtl\"" Oct 30 19:15:14.644947 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.644929 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:15:14.647073 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.647058 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4k9jz\"" Oct 30 19:15:14.647151 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.647081 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:15:14.647151 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.647094 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:15:14.647151 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.647083 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-46fzl\"" Oct 30 19:15:14.647648 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.647633 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:15:14.647689 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.647633 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:15:14.649355 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.649341 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:15:14.649440 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.649367 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:15:14.649440 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.649370 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:15:14.649587 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.649575 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:15:14.650049 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.650027 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:15:14.650172 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.650162 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:15:14.650214 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.650198 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bcb87" Oct 30 19:15:14.660277 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-ovnkube-config\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.660457 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660286 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-host\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.660457 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660319 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-kubelet\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.660457 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660404 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-host\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.660628 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-sys-fs\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.660628 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-tmp-dir\") pod \"node-resolver-25zlr\" (UID: \"ddd92762-f7c4-4f85-80e1-0fe7045e78e8\") " pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:14.660628 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660539 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-sys-fs\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.660628 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-run-netns\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.660829 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660681 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-run-netns\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.660829 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-systemd\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.660829 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660748 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-systemd\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.660829 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660756 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-tmp-dir\") pod \"node-resolver-25zlr\" (UID: \"ddd92762-f7c4-4f85-80e1-0fe7045e78e8\") " pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:14.660829 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysconfig\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.660829 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660802 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-var-lib-kubelet\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.660829 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660807 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysconfig\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.661194 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-systemd-units\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.661194 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660852 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-var-lib-kubelet\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.661194 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysctl-d\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.661194 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660881 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-systemd-units\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.661194 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660903 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-lib-modules\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.661194 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660925 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.661194 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-cni-bin\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.661194 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660963 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-etc-kubernetes\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.661194 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.660975 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysctl-d\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.661194 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661002 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-ovnkube-config\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.661194 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661038 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-tuned\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.661194 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661089 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd91ed10-7ecf-415f-9906-3a79c23c8229-tmp\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.661194 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661117 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-lib-modules\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.661194 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad299622-53d4-4610-b03a-3c586c0fff56-konnectivity-ca\") pod \"konnectivity-agent-lv4mc\" (UID: \"ad299622-53d4-4610-b03a-3c586c0fff56\") " pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-ovn\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661283 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-cnibin\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661307 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-ovn\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysctl-conf\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661327 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661319 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661365 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-sys\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdrqv\" (UniqueName: \"kubernetes.io/projected/bd91ed10-7ecf-415f-9906-3a79c23c8229-kube-api-access-wdrqv\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661400 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-hosts-file\") pod \"node-resolver-25zlr\" (UID: \"ddd92762-f7c4-4f85-80e1-0fe7045e78e8\") " pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661417 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a62fb314-b483-4dcd-8170-a818757da247-ovn-node-metrics-cert\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661437 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-k8s-cni-cncf-io\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661452 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-sys\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjmzh\" (UniqueName: \"kubernetes.io/projected/2293549b-8f94-4bf2-8755-e83e4350fd79-kube-api-access-jjmzh\") pod \"iptables-alerter-lzl7q\" (UID: \"2293549b-8f94-4bf2-8755-e83e4350fd79\") " pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-etc-selinux\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661573 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysctl-conf\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-run\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.661896 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661658 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad299622-53d4-4610-b03a-3c586c0fff56-konnectivity-ca\") pod \"konnectivity-agent-lv4mc\" (UID: \"ad299622-53d4-4610-b03a-3c586c0fff56\") " pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661659 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-dbus\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661756 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-etc-selinux\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661761 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-cni-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661812 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-run\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661822 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/373b617b-0896-420c-b2fd-38635c62a21c-serviceca\") pod \"node-ca-s4k9z\" (UID: \"373b617b-0896-420c-b2fd-38635c62a21c\") " pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661835 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-dbus\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661868 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-systemd\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661866 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-hosts-file\") pod \"node-resolver-25zlr\" (UID: \"ddd92762-f7c4-4f85-80e1-0fe7045e78e8\") " pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661910 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-system-cni-dir\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661921 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-systemd\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.661952 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662001 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-slash\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662046 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662082 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-slash\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.662756 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662106 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2293549b-8f94-4bf2-8755-e83e4350fd79-iptables-alerter-script\") pod \"iptables-alerter-lzl7q\" (UID: \"2293549b-8f94-4bf2-8755-e83e4350fd79\") " pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:14.663582 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-node-log\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.663582 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662159 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-etc-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.663582 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:14.662180 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:14.663582 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5xm\" (UniqueName: \"kubernetes.io/projected/a62fb314-b483-4dcd-8170-a818757da247-kube-api-access-5r5xm\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.663582 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662214 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.663582 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:14.662247 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret podName:ed1dc05f-2a13-4dd2-9d20-f5e96f17099f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:15.162225762 +0000 UTC m=+3.118432914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret") pod "global-pull-secret-syncer-6vks7" (UID: "ed1dc05f-2a13-4dd2-9d20-f5e96f17099f") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:14.663582 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662247 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-node-log\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.663582 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662279 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-etc-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.663582 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662285 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/373b617b-0896-420c-b2fd-38635c62a21c-serviceca\") pod \"node-ca-s4k9z\" (UID: \"373b617b-0896-420c-b2fd-38635c62a21c\") " pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:14.663582 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662345 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7xhf\" (UniqueName: \"kubernetes.io/projected/dc918fe6-f902-419e-b4f1-04690e999f47-kube-api-access-q7xhf\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.663582 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662440 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ks47\" (UniqueName: \"kubernetes.io/projected/a619b2e9-c604-4179-aa69-aa822f209f0c-kube-api-access-9ks47\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:14.663582 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662498 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-hostroot\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.663582 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v974s\" (UniqueName: \"kubernetes.io/projected/373b617b-0896-420c-b2fd-38635c62a21c-kube-api-access-v974s\") pod \"node-ca-s4k9z\" (UID: \"373b617b-0896-420c-b2fd-38635c62a21c\") " pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:14.663582 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662567 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-device-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.663582 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6245n\" (UniqueName: \"kubernetes.io/projected/42b67c56-566b-4319-93ff-165d6cefdf19-kube-api-access-6245n\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.663582 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnzd6\" (UniqueName: \"kubernetes.io/projected/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-kube-api-access-qnzd6\") pod \"node-resolver-25zlr\" (UID: \"ddd92762-f7c4-4f85-80e1-0fe7045e78e8\") " pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-env-overrides\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-modprobe-d\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662678 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-device-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662701 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-cnibin\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662831 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hlgt\" (UniqueName: \"kubernetes.io/projected/a6f66f51-48bd-439d-92b7-a3c9dc33f591-kube-api-access-2hlgt\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-cni-netd\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662889 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-ovnkube-script-lib\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662916 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-system-cni-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662929 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-modprobe-d\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-multus-certs\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.662962 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-cni-netd\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad299622-53d4-4610-b03a-3c586c0fff56-agent-certs\") pod \"konnectivity-agent-lv4mc\" (UID: \"ad299622-53d4-4610-b03a-3c586c0fff56\") " pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663116 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-env-overrides\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663127 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-registration-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663266 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663283 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-registration-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.664315 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663339 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-cni-bin\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663367 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-kubernetes\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663402 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663459 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-cni-bin\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663472 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-kubernetes\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663477 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-ovnkube-script-lib\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663492 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663530 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6f66f51-48bd-439d-92b7-a3c9dc33f591-cni-binary-copy\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/373b617b-0896-420c-b2fd-38635c62a21c-host\") pod \"node-ca-s4k9z\" (UID: \"373b617b-0896-420c-b2fd-38635c62a21c\") " pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-socket-dir-parent\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663631 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-conf-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663643 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663675 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/373b617b-0896-420c-b2fd-38635c62a21c-host\") pod \"node-ca-s4k9z\" (UID: \"373b617b-0896-420c-b2fd-38635c62a21c\") " pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663674 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-daemon-config\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgrg7\" (UniqueName: \"kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7\") pod \"network-check-target-ql75b\" (UID: \"3d1a766f-5256-4044-98d5-f551f811e684\") " pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:14.664805 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663754 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-socket-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.665347 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-log-socket\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.665347 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663809 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-cni-multus\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.665347 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663838 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-var-lib-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.665347 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-kubelet-config\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:14.665347 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663874 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-socket-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.665347 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663891 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-netns\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.665347 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663914 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2293549b-8f94-4bf2-8755-e83e4350fd79-host-slash\") pod \"iptables-alerter-lzl7q\" (UID: \"2293549b-8f94-4bf2-8755-e83e4350fd79\") " pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:14.665347 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663942 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-var-lib-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.665347 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663966 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-log-socket\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.665347 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.663970 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-kubelet-config\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:14.665347 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.664013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-kubelet\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.665347 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.664057 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-kubelet\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.665739 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.665378 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd91ed10-7ecf-415f-9906-3a79c23c8229-tmp\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.665739 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.665413 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-tuned\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.665739 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.665566 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a62fb314-b483-4dcd-8170-a818757da247-ovn-node-metrics-cert\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.665739 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.665707 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad299622-53d4-4610-b03a-3c586c0fff56-agent-certs\") pod \"konnectivity-agent-lv4mc\" (UID: \"ad299622-53d4-4610-b03a-3c586c0fff56\") " pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:15:14.723905 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.723822 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdrqv\" (UniqueName: \"kubernetes.io/projected/bd91ed10-7ecf-415f-9906-3a79c23c8229-kube-api-access-wdrqv\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.724039 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.723927 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnzd6\" (UniqueName: \"kubernetes.io/projected/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-kube-api-access-qnzd6\") pod \"node-resolver-25zlr\" (UID: \"ddd92762-f7c4-4f85-80e1-0fe7045e78e8\") " pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:14.724039 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.723935 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r5xm\" (UniqueName: \"kubernetes.io/projected/a62fb314-b483-4dcd-8170-a818757da247-kube-api-access-5r5xm\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.724039 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.724029 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v974s\" (UniqueName: \"kubernetes.io/projected/373b617b-0896-420c-b2fd-38635c62a21c-kube-api-access-v974s\") pod \"node-ca-s4k9z\" (UID: \"373b617b-0896-420c-b2fd-38635c62a21c\") " pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:14.724165 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.724044 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6245n\" (UniqueName: \"kubernetes.io/projected/42b67c56-566b-4319-93ff-165d6cefdf19-kube-api-access-6245n\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.765249 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-cnibin\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.765249 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765245 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-k8s-cni-cncf-io\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.765482 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765295 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-k8s-cni-cncf-io\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.765482 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765327 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-cnibin\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.765482 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765362 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjmzh\" (UniqueName: \"kubernetes.io/projected/2293549b-8f94-4bf2-8755-e83e4350fd79-kube-api-access-jjmzh\") pod \"iptables-alerter-lzl7q\" (UID: \"2293549b-8f94-4bf2-8755-e83e4350fd79\") " pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:14.765482 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765394 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-cni-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.765482 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-system-cni-dir\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.765482 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765430 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.765482 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2293549b-8f94-4bf2-8755-e83e4350fd79-iptables-alerter-script\") pod \"iptables-alerter-lzl7q\" (UID: \"2293549b-8f94-4bf2-8755-e83e4350fd79\") " pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:14.765482 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765481 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-cni-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.765836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.765836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765507 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7xhf\" (UniqueName: \"kubernetes.io/projected/dc918fe6-f902-419e-b4f1-04690e999f47-kube-api-access-q7xhf\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.765836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765480 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-system-cni-dir\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.765836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765573 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ks47\" (UniqueName: \"kubernetes.io/projected/a619b2e9-c604-4179-aa69-aa822f209f0c-kube-api-access-9ks47\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:14.765836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765603 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-hostroot\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.765836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-cnibin\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.765836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hlgt\" (UniqueName: \"kubernetes.io/projected/a6f66f51-48bd-439d-92b7-a3c9dc33f591-kube-api-access-2hlgt\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.765836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-system-cni-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.765836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765704 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-hostroot\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.765836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765719 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-multus-certs\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.765836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765754 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.765836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765773 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-cnibin\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.765836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765784 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:14.765836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6f66f51-48bd-439d-92b7-a3c9dc33f591-cni-binary-copy\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.765836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765839 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-system-cni-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.765836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-socket-dir-parent\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-conf-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765897 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-socket-dir-parent\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-daemon-config\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrg7\" (UniqueName: \"kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7\") pod \"network-check-target-ql75b\" (UID: \"3d1a766f-5256-4044-98d5-f551f811e684\") " pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-cni-multus\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:14.765964 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.765988 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-netns\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766014 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2293549b-8f94-4bf2-8755-e83e4350fd79-host-slash\") pod \"iptables-alerter-lzl7q\" (UID: \"2293549b-8f94-4bf2-8755-e83e4350fd79\") " pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:14.766063 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs podName:a619b2e9-c604-4179-aa69-aa822f209f0c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:15.266033396 +0000 UTC m=+3.222240546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs") pod "network-metrics-daemon-7gj92" (UID: "a619b2e9-c604-4179-aa69-aa822f209f0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-kubelet\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766128 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-cni-bin\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766179 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-etc-kubernetes\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766209 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2293549b-8f94-4bf2-8755-e83e4350fd79-iptables-alerter-script\") pod \"iptables-alerter-lzl7q\" (UID: \"2293549b-8f94-4bf2-8755-e83e4350fd79\") " pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766239 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766251 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.766553 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766265 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2293549b-8f94-4bf2-8755-e83e4350fd79-host-slash\") pod \"iptables-alerter-lzl7q\" (UID: \"2293549b-8f94-4bf2-8755-e83e4350fd79\") " pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:14.767114 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766255 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-etc-kubernetes\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.767114 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766285 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-conf-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.767114 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766316 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-kubelet\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.767114 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766316 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-netns\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.767114 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766012 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-multus-certs\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.767114 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766355 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-cni-multus\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.767114 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766356 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-cni-bin\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.767114 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766445 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.767114 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766561 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6f66f51-48bd-439d-92b7-a3c9dc33f591-cni-binary-copy\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.767114 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766565 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-daemon-config\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.767114 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.766689 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.824409 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:14.824377 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:15:14.824409 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:14.824403 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:15:14.824409 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:14.824417 2577 projected.go:196] Error preparing data for projected volume kube-api-access-fgrg7 for pod openshift-network-diagnostics/network-check-target-ql75b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:14.824699 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:14.824484 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7 podName:3d1a766f-5256-4044-98d5-f551f811e684 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:15.324467166 +0000 UTC m=+3.280674311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fgrg7" (UniqueName: "kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7") pod "network-check-target-ql75b" (UID: "3d1a766f-5256-4044-98d5-f551f811e684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:14.826432 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.826403 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7xhf\" (UniqueName: \"kubernetes.io/projected/dc918fe6-f902-419e-b4f1-04690e999f47-kube-api-access-q7xhf\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.826432 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.826419 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjmzh\" (UniqueName: \"kubernetes.io/projected/2293549b-8f94-4bf2-8755-e83e4350fd79-kube-api-access-jjmzh\") pod \"iptables-alerter-lzl7q\" (UID: \"2293549b-8f94-4bf2-8755-e83e4350fd79\") " pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:14.826642 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.826623 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hlgt\" (UniqueName: \"kubernetes.io/projected/a6f66f51-48bd-439d-92b7-a3c9dc33f591-kube-api-access-2hlgt\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.826730 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.826711 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ks47\" (UniqueName: \"kubernetes.io/projected/a619b2e9-c604-4179-aa69-aa822f209f0c-kube-api-access-9ks47\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:14.845487 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.845456 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:15:14.856455 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:14.856426 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad299622_53d4_4610_b03a_3c586c0fff56.slice/crio-a4f67617ed451d37c9bee2fa996dcac9e63113d20cb96ca12df1d0334d934b85 WatchSource:0}: Error finding container a4f67617ed451d37c9bee2fa996dcac9e63113d20cb96ca12df1d0334d934b85: Status 404 returned error can't find the container with id a4f67617ed451d37c9bee2fa996dcac9e63113d20cb96ca12df1d0334d934b85 Oct 30 19:15:14.858189 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.858161 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:14.862143 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.862127 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:14.866886 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:14.866865 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b67c56_566b_4319_93ff_165d6cefdf19.slice/crio-cc8168a8fa89b0a7de21570865493f7d8545df2b6a770930a417edb4cffa8676 WatchSource:0}: Error finding container cc8168a8fa89b0a7de21570865493f7d8545df2b6a770930a417edb4cffa8676: Status 404 returned error can't find the container with id cc8168a8fa89b0a7de21570865493f7d8545df2b6a770930a417edb4cffa8676 Oct 30 19:15:14.871844 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:14.871821 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd92762_f7c4_4f85_80e1_0fe7045e78e8.slice/crio-00f7c9795076d2276b5d53c2f77ece72abdba65e4e5b47dcd39cd0350e710ff8 WatchSource:0}: Error finding container 00f7c9795076d2276b5d53c2f77ece72abdba65e4e5b47dcd39cd0350e710ff8: Status 404 returned error can't find the container with id 00f7c9795076d2276b5d53c2f77ece72abdba65e4e5b47dcd39cd0350e710ff8 Oct 30 19:15:14.876222 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.876204 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:14.878364 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.878348 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:14.884451 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.884425 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:14.886071 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:14.886021 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd91ed10_7ecf_415f_9906_3a79c23c8229.slice/crio-1fe6bbf253d0de0333de97ad32765790f872c846bd1790f3c4ba72cbf77451d4 WatchSource:0}: Error finding container 1fe6bbf253d0de0333de97ad32765790f872c846bd1790f3c4ba72cbf77451d4: Status 404 returned error can't find the container with id 1fe6bbf253d0de0333de97ad32765790f872c846bd1790f3c4ba72cbf77451d4 Oct 30 19:15:14.886924 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.886868 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:14.889395 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:14.889366 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373b617b_0896_420c_b2fd_38635c62a21c.slice/crio-de85c47ff6f88560baa95f2819c00fade8d284c35087bf8ad98102c40cdb279f WatchSource:0}: Error finding container de85c47ff6f88560baa95f2819c00fade8d284c35087bf8ad98102c40cdb279f: Status 404 returned error can't find the container with id de85c47ff6f88560baa95f2819c00fade8d284c35087bf8ad98102c40cdb279f Oct 30 19:15:14.894888 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:14.894866 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62fb314_b483_4dcd_8170_a818757da247.slice/crio-cb0314051d7513894134c186b87d00d4f1d98174bb3362a535b1d060c0ccad66 WatchSource:0}: Error finding container cb0314051d7513894134c186b87d00d4f1d98174bb3362a535b1d060c0ccad66: Status 404 returned error can't find the container with id cb0314051d7513894134c186b87d00d4f1d98174bb3362a535b1d060c0ccad66 Oct 30 19:15:14.896867 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:14.896824 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2293549b_8f94_4bf2_8755_e83e4350fd79.slice/crio-1eee746462d68b33b3b02fe2ac8ca0126457fa5e52540af045fe141565c717a5 WatchSource:0}: Error finding container 1eee746462d68b33b3b02fe2ac8ca0126457fa5e52540af045fe141565c717a5: Status 404 returned error can't find the container with id 1eee746462d68b33b3b02fe2ac8ca0126457fa5e52540af045fe141565c717a5 Oct 30 19:15:14.902245 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.902226 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kldq6" Oct 30 19:15:14.906128 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:14.906111 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:14.912701 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:14.912677 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f66f51_48bd_439d_92b7_a3c9dc33f591.slice/crio-832fa605d5ce18b179ff8c4e2c15029b60c9003af1371f9cc5624a3cbb6b1976 WatchSource:0}: Error finding container 832fa605d5ce18b179ff8c4e2c15029b60c9003af1371f9cc5624a3cbb6b1976: Status 404 returned error can't find the container with id 832fa605d5ce18b179ff8c4e2c15029b60c9003af1371f9cc5624a3cbb6b1976 Oct 30 19:15:14.915295 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:14.915274 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc918fe6_f902_419e_b4f1_04690e999f47.slice/crio-d6cac17a243ecfd4611217de16d529f97a77d4c9e00d3fcfa0662a1a1c8cd3f7 WatchSource:0}: Error finding container d6cac17a243ecfd4611217de16d529f97a77d4c9e00d3fcfa0662a1a1c8cd3f7: Status 404 returned error can't find the container with id d6cac17a243ecfd4611217de16d529f97a77d4c9e00d3fcfa0662a1a1c8cd3f7 Oct 30 19:15:15.169692 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:15.169560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:15.169878 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:15.169765 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:15.169878 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:15.169846 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret podName:ed1dc05f-2a13-4dd2-9d20-f5e96f17099f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:16.169824787 +0000 UTC m=+4.126031967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret") pod "global-pull-secret-syncer-6vks7" (UID: "ed1dc05f-2a13-4dd2-9d20-f5e96f17099f") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:15.270375 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:15.270333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:15.270547 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:15.270514 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:15.270607 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:15.270601 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs podName:a619b2e9-c604-4179-aa69-aa822f209f0c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:16.270578138 +0000 UTC m=+4.226785273 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs") pod "network-metrics-daemon-7gj92" (UID: "a619b2e9-c604-4179-aa69-aa822f209f0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:15.371920 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:15.371881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrg7\" (UniqueName: \"kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7\") pod \"network-check-target-ql75b\" (UID: \"3d1a766f-5256-4044-98d5-f551f811e684\") " pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:15.372162 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:15.372039 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:15:15.372162 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:15.372065 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:15:15.372162 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:15.372080 2577 projected.go:196] Error preparing data for projected volume kube-api-access-fgrg7 for pod openshift-network-diagnostics/network-check-target-ql75b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:15.372162 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:15.372164 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7 podName:3d1a766f-5256-4044-98d5-f551f811e684 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:16.372144522 +0000 UTC m=+4.328351668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fgrg7" (UniqueName: "kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7") pod "network-check-target-ql75b" (UID: "3d1a766f-5256-4044-98d5-f551f811e684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:15.616132 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:15.616026 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-25zlr" event={"ID":"ddd92762-f7c4-4f85-80e1-0fe7045e78e8","Type":"ContainerStarted","Data":"00f7c9795076d2276b5d53c2f77ece72abdba65e4e5b47dcd39cd0350e710ff8"} Oct 30 19:15:15.617485 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:15.617446 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" event={"ID":"42b67c56-566b-4319-93ff-165d6cefdf19","Type":"ContainerStarted","Data":"cc8168a8fa89b0a7de21570865493f7d8545df2b6a770930a417edb4cffa8676"} Oct 30 19:15:15.618834 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:15.618802 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lv4mc" event={"ID":"ad299622-53d4-4610-b03a-3c586c0fff56","Type":"ContainerStarted","Data":"a4f67617ed451d37c9bee2fa996dcac9e63113d20cb96ca12df1d0334d934b85"} Oct 30 19:15:15.620086 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:15.620052 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2hmn" event={"ID":"dc918fe6-f902-419e-b4f1-04690e999f47","Type":"ContainerStarted","Data":"d6cac17a243ecfd4611217de16d529f97a77d4c9e00d3fcfa0662a1a1c8cd3f7"} Oct 30 19:15:15.621504 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:15.621480 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s4k9z" event={"ID":"373b617b-0896-420c-b2fd-38635c62a21c","Type":"ContainerStarted","Data":"de85c47ff6f88560baa95f2819c00fade8d284c35087bf8ad98102c40cdb279f"} Oct 30 19:15:15.623015 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:15.622966 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8ch84" event={"ID":"bd91ed10-7ecf-415f-9906-3a79c23c8229","Type":"ContainerStarted","Data":"1fe6bbf253d0de0333de97ad32765790f872c846bd1790f3c4ba72cbf77451d4"} Oct 30 19:15:15.624291 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:15.624265 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kldq6" event={"ID":"a6f66f51-48bd-439d-92b7-a3c9dc33f591","Type":"ContainerStarted","Data":"832fa605d5ce18b179ff8c4e2c15029b60c9003af1371f9cc5624a3cbb6b1976"} Oct 30 19:15:15.625713 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:15.625662 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lzl7q" event={"ID":"2293549b-8f94-4bf2-8755-e83e4350fd79","Type":"ContainerStarted","Data":"1eee746462d68b33b3b02fe2ac8ca0126457fa5e52540af045fe141565c717a5"} Oct 30 19:15:15.627533 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:15.627504 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" event={"ID":"a62fb314-b483-4dcd-8170-a818757da247","Type":"ContainerStarted","Data":"cb0314051d7513894134c186b87d00d4f1d98174bb3362a535b1d060c0ccad66"} Oct 30 19:15:15.653066 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:15.652994 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:10:15 +0000 UTC" deadline="2028-06-17 11:43:04.182641954 +0000 UTC" Oct 30 19:15:15.653066 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:15.653058 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14296h27m48.529588909s" Oct 30 19:15:16.178312 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:16.178271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:16.178533 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:16.178468 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:16.178595 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:16.178537 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret podName:ed1dc05f-2a13-4dd2-9d20-f5e96f17099f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:18.178515208 +0000 UTC m=+6.134722357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret") pod "global-pull-secret-syncer-6vks7" (UID: "ed1dc05f-2a13-4dd2-9d20-f5e96f17099f") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:16.279498 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:16.279460 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:16.279756 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:16.279639 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:16.279756 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:16.279709 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs podName:a619b2e9-c604-4179-aa69-aa822f209f0c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:18.279689097 +0000 UTC m=+6.235896243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs") pod "network-metrics-daemon-7gj92" (UID: "a619b2e9-c604-4179-aa69-aa822f209f0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:16.380393 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:16.380349 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrg7\" (UniqueName: \"kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7\") pod \"network-check-target-ql75b\" (UID: \"3d1a766f-5256-4044-98d5-f551f811e684\") " pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:16.380596 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:16.380581 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:15:16.380659 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:16.380607 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:15:16.380659 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:16.380621 2577 projected.go:196] Error preparing data for projected volume kube-api-access-fgrg7 for pod openshift-network-diagnostics/network-check-target-ql75b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:16.380785 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:16.380686 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7 podName:3d1a766f-5256-4044-98d5-f551f811e684 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:18.380666477 +0000 UTC m=+6.336873624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fgrg7" (UniqueName: "kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7") pod "network-check-target-ql75b" (UID: "3d1a766f-5256-4044-98d5-f551f811e684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:16.610627 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:16.609875 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:16.610627 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:16.610008 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ql75b" podUID="3d1a766f-5256-4044-98d5-f551f811e684" Oct 30 19:15:16.610627 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:16.610299 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:16.610627 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:16.610416 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7gj92" podUID="a619b2e9-c604-4179-aa69-aa822f209f0c" Oct 30 19:15:16.610627 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:16.610478 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:16.610627 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:16.610552 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vks7" podUID="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" Oct 30 19:15:16.653952 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:16.653907 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:10:15 +0000 UTC" deadline="2028-05-06 21:56:31.269112769 +0000 UTC" Oct 30 19:15:16.653952 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:16.653951 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13298h41m14.615166628s" Oct 30 19:15:18.198994 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:18.198959 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:18.199585 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:18.199147 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:18.199585 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:18.199237 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret podName:ed1dc05f-2a13-4dd2-9d20-f5e96f17099f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:22.199215551 +0000 UTC m=+10.155422699 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret") pod "global-pull-secret-syncer-6vks7" (UID: "ed1dc05f-2a13-4dd2-9d20-f5e96f17099f") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:18.299392 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:18.299347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:18.299593 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:18.299574 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:18.299661 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:18.299647 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs podName:a619b2e9-c604-4179-aa69-aa822f209f0c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:22.299626396 +0000 UTC m=+10.255833545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs") pod "network-metrics-daemon-7gj92" (UID: "a619b2e9-c604-4179-aa69-aa822f209f0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:18.400530 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:18.400130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrg7\" (UniqueName: \"kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7\") pod \"network-check-target-ql75b\" (UID: \"3d1a766f-5256-4044-98d5-f551f811e684\") " pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:18.400530 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:18.400374 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:15:18.400530 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:18.400397 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:15:18.400530 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:18.400413 2577 projected.go:196] Error preparing data for projected volume kube-api-access-fgrg7 for pod openshift-network-diagnostics/network-check-target-ql75b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:18.400530 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:18.400479 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7 podName:3d1a766f-5256-4044-98d5-f551f811e684 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:22.400460344 +0000 UTC m=+10.356667493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fgrg7" (UniqueName: "kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7") pod "network-check-target-ql75b" (UID: "3d1a766f-5256-4044-98d5-f551f811e684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:18.610317 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:18.610219 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:18.610317 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:18.610283 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:18.610317 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:18.610297 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:18.610604 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:18.610385 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ql75b" podUID="3d1a766f-5256-4044-98d5-f551f811e684" Oct 30 19:15:18.610604 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:18.610465 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7gj92" podUID="a619b2e9-c604-4179-aa69-aa822f209f0c" Oct 30 19:15:18.610604 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:18.610511 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vks7" podUID="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" Oct 30 19:15:18.663439 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:18.662417 2577 generic.go:356] "Generic (PLEG): container finished" podID="56c484cac7643547da8de4853003de0e" containerID="3ee51a9598682182e4d6661d583ea5d95b56b6ea1fd1412b0e80957b5ccfd965" exitCode=0 Oct 30 19:15:18.663439 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:18.662521 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" event={"ID":"56c484cac7643547da8de4853003de0e","Type":"ContainerDied","Data":"3ee51a9598682182e4d6661d583ea5d95b56b6ea1fd1412b0e80957b5ccfd965"} Oct 30 19:15:18.679072 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:18.678524 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" event={"ID":"d17eaea73d7bf32a8a9bb9e63451f939","Type":"ContainerStarted","Data":"c0b36c4262aca8f337d1546c449c83549d970509e0285b699d352906e971194d"} Oct 30 19:15:19.692454 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:19.692411 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" event={"ID":"56c484cac7643547da8de4853003de0e","Type":"ContainerStarted","Data":"772b2159470cd609c9ded161021dea72191c1d9016efa1d580987bda83459b5d"} Oct 30 19:15:19.696222 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:19.696187 2577 generic.go:356] "Generic (PLEG): container finished" podID="dc918fe6-f902-419e-b4f1-04690e999f47" containerID="c276a97839d8a209e11376eaec34037f86127aaec5b3f2a49e5e2a6e94149650" exitCode=0 Oct 30 19:15:19.697195 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:19.697129 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2hmn" event={"ID":"dc918fe6-f902-419e-b4f1-04690e999f47","Type":"ContainerDied","Data":"c276a97839d8a209e11376eaec34037f86127aaec5b3f2a49e5e2a6e94149650"} Oct 30 19:15:19.816958 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:19.816324 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" podStartSLOduration=5.816303452 podStartE2EDuration="5.816303452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:18.90881632 +0000 UTC m=+6.865023475" watchObservedRunningTime="2026-10-30 19:15:19.816303452 +0000 UTC m=+7.772510608" Oct 30 19:15:19.816958 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:19.816628 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" podStartSLOduration=5.816619925 podStartE2EDuration="5.816619925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:19.815419079 +0000 UTC m=+7.771626235" watchObservedRunningTime="2026-10-30 19:15:19.816619925 +0000 UTC m=+7.772827083" Oct 30 19:15:20.610196 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:20.610158 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:20.610196 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:20.610189 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:20.610441 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:20.610300 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vks7" podUID="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" Oct 30 19:15:20.610441 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:20.610376 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ql75b" podUID="3d1a766f-5256-4044-98d5-f551f811e684" Oct 30 19:15:20.610441 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:20.610410 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:20.610596 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:20.610470 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7gj92" podUID="a619b2e9-c604-4179-aa69-aa822f209f0c" Oct 30 19:15:20.708476 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:20.708443 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lv4mc" event={"ID":"ad299622-53d4-4610-b03a-3c586c0fff56","Type":"ContainerStarted","Data":"dd70a55b9bc29d5ee1ccf8580e40cfcd5587ff710235bd5608f631917623b831"} Oct 30 19:15:20.831968 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:20.831906 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lv4mc" podStartSLOduration=2.972310018 podStartE2EDuration="7.831886906s" totalImagesPullingTime="4.859576888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:20.831123541 +0000 UTC m=+8.787330703" watchObservedRunningTime="2026-10-30 19:15:20.831886906 +0000 UTC m=+8.788094061" Oct 30 19:15:21.453208 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:21.453048 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:15:21.453919 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:21.453688 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:15:21.714609 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:21.714306 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s4k9z" event={"ID":"373b617b-0896-420c-b2fd-38635c62a21c","Type":"ContainerStarted","Data":"25c2cdb14ddd121cc627299bbc1285f45984402b45ddf215579a6cd90764fb53"} Oct 30 19:15:21.837223 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:21.837163 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s4k9z" podStartSLOduration=3.055499952 podStartE2EDuration="8.837143804s" totalImagesPullingTime="5.781643852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:21.83683658 +0000 UTC m=+9.793043736" watchObservedRunningTime="2026-10-30 19:15:21.837143804 +0000 UTC m=+9.793350961" Oct 30 19:15:22.234749 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:22.234661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:22.234944 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:22.234877 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:22.235039 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:22.234949 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret podName:ed1dc05f-2a13-4dd2-9d20-f5e96f17099f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:30.234928132 +0000 UTC m=+18.191135270 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret") pod "global-pull-secret-syncer-6vks7" (UID: "ed1dc05f-2a13-4dd2-9d20-f5e96f17099f") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:22.335745 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:22.335685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:22.336000 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:22.335859 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:22.336000 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:22.335913 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs podName:a619b2e9-c604-4179-aa69-aa822f209f0c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:30.335899095 +0000 UTC m=+18.292106226 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs") pod "network-metrics-daemon-7gj92" (UID: "a619b2e9-c604-4179-aa69-aa822f209f0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:22.437029 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:22.436950 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrg7\" (UniqueName: \"kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7\") pod \"network-check-target-ql75b\" (UID: \"3d1a766f-5256-4044-98d5-f551f811e684\") " pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:22.437241 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:22.437167 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:15:22.437241 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:22.437189 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:15:22.437241 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:22.437202 2577 projected.go:196] Error preparing data for projected volume kube-api-access-fgrg7 for pod openshift-network-diagnostics/network-check-target-ql75b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:22.437419 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:22.437274 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7 podName:3d1a766f-5256-4044-98d5-f551f811e684 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:30.437252429 +0000 UTC m=+18.393459562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fgrg7" (UniqueName: "kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7") pod "network-check-target-ql75b" (UID: "3d1a766f-5256-4044-98d5-f551f811e684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:22.611072 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:22.609816 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:22.611072 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:22.609955 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vks7" podUID="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" Oct 30 19:15:22.611072 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:22.610025 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:22.611072 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:22.610115 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ql75b" podUID="3d1a766f-5256-4044-98d5-f551f811e684" Oct 30 19:15:22.611072 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:22.610181 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:22.611072 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:22.610269 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7gj92" podUID="a619b2e9-c604-4179-aa69-aa822f209f0c" Oct 30 19:15:22.718358 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:22.718267 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" event={"ID":"42b67c56-566b-4319-93ff-165d6cefdf19","Type":"ContainerStarted","Data":"c319bd70b6f5e8bcec1f6251bed48699618851e51b5e32fac78d42490291f4a9"} Oct 30 19:15:22.718870 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:22.718840 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:15:24.132752 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:24.132721 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:15:24.525199 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:24.525100 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:15:24.13274832Z","UUID":"8e9b75b9-3e5d-4263-bead-56ef5d534990","Handler":null,"Name":"","Endpoint":""} Oct 30 19:15:24.527063 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:24.526542 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:15:24.527063 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:24.526628 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:15:24.608372 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:24.608330 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:24.608558 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:24.608436 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:24.608617 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:24.608559 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ql75b" podUID="3d1a766f-5256-4044-98d5-f551f811e684" Oct 30 19:15:24.608617 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:24.608604 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:24.608710 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:24.608683 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7gj92" podUID="a619b2e9-c604-4179-aa69-aa822f209f0c" Oct 30 19:15:24.608768 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:24.608756 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vks7" podUID="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" Oct 30 19:15:24.725068 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:24.725023 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-25zlr" event={"ID":"ddd92762-f7c4-4f85-80e1-0fe7045e78e8","Type":"ContainerStarted","Data":"5baedba82503893da742500f40c9d24bf30c04ec9ce8ba7d3f35bff219f2f9e0"} Oct 30 19:15:24.727141 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:24.727108 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" event={"ID":"42b67c56-566b-4319-93ff-165d6cefdf19","Type":"ContainerStarted","Data":"e045e3adf3cf8a1e88faf8f7d11d10ba59862bf8f45fcbf4e55e25e862a34eac"} Oct 30 19:15:24.729222 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:24.729193 2577 generic.go:356] "Generic (PLEG): container finished" podID="dc918fe6-f902-419e-b4f1-04690e999f47" containerID="0f50efc1235efc95c66811ead4939c8278d38048d262900e9e294a3cbc3ceda0" exitCode=0 Oct 30 19:15:24.729344 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:24.729284 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2hmn" event={"ID":"dc918fe6-f902-419e-b4f1-04690e999f47","Type":"ContainerDied","Data":"0f50efc1235efc95c66811ead4939c8278d38048d262900e9e294a3cbc3ceda0"} Oct 30 19:15:24.731022 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:24.730997 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kldq6" event={"ID":"a6f66f51-48bd-439d-92b7-a3c9dc33f591","Type":"ContainerStarted","Data":"eefe939fe93fae41641fb1c0c873c5c217d6643261cbe4dbbcfa2ce70a98a1d5"} Oct 30 19:15:24.837525 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:24.837466 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-25zlr" podStartSLOduration=2.462435282 podStartE2EDuration="11.837451989s" totalImagesPullingTime="9.375016707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:24.837430291 +0000 UTC m=+12.793637443" watchObservedRunningTime="2026-10-30 19:15:24.837451989 +0000 UTC m=+12.793659142" Oct 30 19:15:25.069329 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:25.069230 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-kldq6" podStartSLOduration=2.145004534 podStartE2EDuration="11.069214061s" totalImagesPullingTime="8.924209527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:25.0688696 +0000 UTC m=+13.025076751" watchObservedRunningTime="2026-10-30 19:15:25.069214061 +0000 UTC m=+13.025421213" Oct 30 19:15:25.736631 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:25.736335 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lzl7q" event={"ID":"2293549b-8f94-4bf2-8755-e83e4350fd79","Type":"ContainerStarted","Data":"c6f1814e89c4cf1334760c08a3d3c8f3ad0107ae88842ada4811e8445d59d891"} Oct 30 19:15:25.855240 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:25.855170 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lzl7q" podStartSLOduration=2.894026012 podStartE2EDuration="11.855151752s" totalImagesPullingTime="8.96112574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:25.854785896 +0000 UTC m=+13.810993048" watchObservedRunningTime="2026-10-30 19:15:25.855151752 +0000 UTC m=+13.811358904" Oct 30 19:15:26.608150 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:26.608111 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:26.608328 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:26.608154 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:26.608328 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:26.608250 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vks7" podUID="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" Oct 30 19:15:26.608328 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:26.608294 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:26.608456 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:26.608384 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ql75b" podUID="3d1a766f-5256-4044-98d5-f551f811e684" Oct 30 19:15:26.608502 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:26.608459 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7gj92" podUID="a619b2e9-c604-4179-aa69-aa822f209f0c" Oct 30 19:15:26.740535 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:26.740443 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" event={"ID":"42b67c56-566b-4319-93ff-165d6cefdf19","Type":"ContainerStarted","Data":"1357a85748a6804a18c21339eea89a58713d08a900f9c3642baeb20c9bc78107"} Oct 30 19:15:26.742958 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:26.742929 2577 generic.go:356] "Generic (PLEG): container finished" podID="dc918fe6-f902-419e-b4f1-04690e999f47" containerID="0ff215eceaa445823ea8225f39894e2688ac507b5d9aa004cc3864259a662f64" exitCode=0 Oct 30 19:15:26.743104 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:26.743023 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2hmn" event={"ID":"dc918fe6-f902-419e-b4f1-04690e999f47","Type":"ContainerDied","Data":"0ff215eceaa445823ea8225f39894e2688ac507b5d9aa004cc3864259a662f64"} Oct 30 19:15:26.744888 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:26.744779 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8ch84" event={"ID":"bd91ed10-7ecf-415f-9906-3a79c23c8229","Type":"ContainerStarted","Data":"9c112144aa812ffebb747c974c14f316a1f5c988ee02c21872c9fc303e5c48b6"} Oct 30 19:15:26.861480 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:26.861425 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" podStartSLOduration=3.025421376 podStartE2EDuration="13.861410835s" totalImagesPullingTime="10.835989459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:26.860953318 +0000 UTC m=+14.817160470" watchObservedRunningTime="2026-10-30 19:15:26.861410835 +0000 UTC m=+14.817617991" Oct 30 19:15:27.095194 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:27.095132 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8ch84" podStartSLOduration=2.873272455 podStartE2EDuration="14.095117089s" totalImagesPullingTime="11.221844634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:27.094956845 +0000 UTC m=+15.051163998" watchObservedRunningTime="2026-10-30 19:15:27.095117089 +0000 UTC m=+15.051324240" Oct 30 19:15:28.607764 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:28.607736 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:28.608395 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:28.607736 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:28.608395 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:28.607854 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7gj92" podUID="a619b2e9-c604-4179-aa69-aa822f209f0c" Oct 30 19:15:28.608395 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:28.607866 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:28.608395 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:28.607997 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vks7" podUID="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" Oct 30 19:15:28.608395 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:28.608041 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ql75b" podUID="3d1a766f-5256-4044-98d5-f551f811e684" Oct 30 19:15:28.751798 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:28.751704 2577 generic.go:356] "Generic (PLEG): container finished" podID="dc918fe6-f902-419e-b4f1-04690e999f47" containerID="1b083fc73a549a823599cfaa2c37af0c09e5e12dbd3341f36d723af3b2352689" exitCode=0 Oct 30 19:15:28.751798 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:28.751771 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2hmn" event={"ID":"dc918fe6-f902-419e-b4f1-04690e999f47","Type":"ContainerDied","Data":"1b083fc73a549a823599cfaa2c37af0c09e5e12dbd3341f36d723af3b2352689"} Oct 30 19:15:30.302176 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:30.301937 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:30.302629 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:30.302098 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:30.302629 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:30.302256 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret podName:ed1dc05f-2a13-4dd2-9d20-f5e96f17099f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:46.302241332 +0000 UTC m=+34.258448465 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret") pod "global-pull-secret-syncer-6vks7" (UID: "ed1dc05f-2a13-4dd2-9d20-f5e96f17099f") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:30.402751 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:30.402718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:30.402918 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:30.402897 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:30.403001 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:30.402990 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs podName:a619b2e9-c604-4179-aa69-aa822f209f0c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:46.402956212 +0000 UTC m=+34.359163358 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs") pod "network-metrics-daemon-7gj92" (UID: "a619b2e9-c604-4179-aa69-aa822f209f0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:30.503928 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:30.503886 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrg7\" (UniqueName: \"kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7\") pod \"network-check-target-ql75b\" (UID: \"3d1a766f-5256-4044-98d5-f551f811e684\") " pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:30.504138 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:30.504053 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:15:30.504138 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:30.504073 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:15:30.504138 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:30.504087 2577 projected.go:196] Error preparing data for projected volume kube-api-access-fgrg7 for pod openshift-network-diagnostics/network-check-target-ql75b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:30.504298 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:30.504143 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7 podName:3d1a766f-5256-4044-98d5-f551f811e684 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:46.504125153 +0000 UTC m=+34.460332296 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fgrg7" (UniqueName: "kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7") pod "network-check-target-ql75b" (UID: "3d1a766f-5256-4044-98d5-f551f811e684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:30.607998 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:30.607946 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:30.607998 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:30.607961 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:30.608223 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:30.608101 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vks7" podUID="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" Oct 30 19:15:30.608223 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:30.608154 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ql75b" podUID="3d1a766f-5256-4044-98d5-f551f811e684" Oct 30 19:15:30.608223 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:30.608182 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:30.608350 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:30.608272 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7gj92" podUID="a619b2e9-c604-4179-aa69-aa822f209f0c" Oct 30 19:15:31.761730 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:31.761534 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" event={"ID":"a62fb314-b483-4dcd-8170-a818757da247","Type":"ContainerStarted","Data":"1c38716c7b94dbd3a9c5bb61f407642328c5df88b26611e1c31d855e942f4e76"} Oct 30 19:15:31.762144 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:31.761821 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" event={"ID":"a62fb314-b483-4dcd-8170-a818757da247","Type":"ContainerStarted","Data":"f8798caf47ee2545d6bfa7576f7dee65efbb79081610a061eb36557c784198e3"} Oct 30 19:15:31.762144 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:31.761871 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" event={"ID":"a62fb314-b483-4dcd-8170-a818757da247","Type":"ContainerStarted","Data":"090c21c8e45ecc3c86cddcf885ea0ab2e21906100bec7ad5d196a48e93920f3f"} Oct 30 19:15:31.762144 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:31.761912 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" event={"ID":"a62fb314-b483-4dcd-8170-a818757da247","Type":"ContainerStarted","Data":"28c6fff6791b8af76b3d35ec161839ef2ee6a3aae4d299bbf0364a953491d0d1"} Oct 30 19:15:31.762144 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:31.761957 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" event={"ID":"a62fb314-b483-4dcd-8170-a818757da247","Type":"ContainerStarted","Data":"00d0ccd89d3d90fd2fd89d9c5f98ae9eac6048fb482d6106ebda63e905f8c2ea"} Oct 30 19:15:31.762144 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:31.762004 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" event={"ID":"a62fb314-b483-4dcd-8170-a818757da247","Type":"ContainerStarted","Data":"a6881e799185a5d614ebfa5c448b35705bcfd32e364cb713a7b066014ac02fe6"} Oct 30 19:15:31.764299 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:31.764275 2577 generic.go:356] "Generic (PLEG): container finished" podID="dc918fe6-f902-419e-b4f1-04690e999f47" containerID="535dde6497a5208aa3f4c9db6f1e1882fdd41b7886e78af6e7d52437a5a21098" exitCode=0 Oct 30 19:15:31.764414 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:31.764307 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2hmn" event={"ID":"dc918fe6-f902-419e-b4f1-04690e999f47","Type":"ContainerDied","Data":"535dde6497a5208aa3f4c9db6f1e1882fdd41b7886e78af6e7d52437a5a21098"} Oct 30 19:15:32.609122 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:32.609085 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:32.609335 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:32.609166 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:32.609335 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:32.609095 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:32.609335 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:32.609171 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ql75b" podUID="3d1a766f-5256-4044-98d5-f551f811e684" Oct 30 19:15:32.609335 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:32.609275 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7gj92" podUID="a619b2e9-c604-4179-aa69-aa822f209f0c" Oct 30 19:15:32.609501 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:32.609358 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vks7" podUID="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" Oct 30 19:15:32.769520 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:32.769485 2577 generic.go:356] "Generic (PLEG): container finished" podID="dc918fe6-f902-419e-b4f1-04690e999f47" containerID="ab085e856154b4612b87cb2ef40f5a932121cc317329c98b545a663cfc1406b3" exitCode=0 Oct 30 19:15:32.769994 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:32.769533 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2hmn" event={"ID":"dc918fe6-f902-419e-b4f1-04690e999f47","Type":"ContainerDied","Data":"ab085e856154b4612b87cb2ef40f5a932121cc317329c98b545a663cfc1406b3"} Oct 30 19:15:33.774447 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:33.774343 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" event={"ID":"a62fb314-b483-4dcd-8170-a818757da247","Type":"ContainerStarted","Data":"8274d68d9c5bd14b4f2bd0d7ac685b033c49fbac0447ed2273ca34879ec74694"} Oct 30 19:15:33.777464 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:33.777436 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2hmn" event={"ID":"dc918fe6-f902-419e-b4f1-04690e999f47","Type":"ContainerStarted","Data":"12433fd32946520fd244856710769361f4face661e9a4ee24be9f8c0f2e048ee"} Oct 30 19:15:33.899745 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:33.899694 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v2hmn" podStartSLOduration=5.753203905 podStartE2EDuration="19.899680292s" totalImagesPullingTime="14.146476387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:33.899085893 +0000 UTC m=+21.855293058" watchObservedRunningTime="2026-10-30 19:15:33.899680292 +0000 UTC m=+21.855887443" Oct 30 19:15:34.608397 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:34.608359 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:34.608397 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:34.608400 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:34.608613 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:34.608488 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ql75b" podUID="3d1a766f-5256-4044-98d5-f551f811e684" Oct 30 19:15:34.608613 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:34.608561 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vks7" podUID="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" Oct 30 19:15:34.608679 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:34.608650 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:34.608757 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:34.608739 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7gj92" podUID="a619b2e9-c604-4179-aa69-aa822f209f0c" Oct 30 19:15:35.786092 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:35.785963 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" event={"ID":"a62fb314-b483-4dcd-8170-a818757da247","Type":"ContainerStarted","Data":"acf3a50be3e315efd9c8e8f298a94c98e723a5453aa696881175274d1562d11d"} Oct 30 19:15:35.787792 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:35.787767 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:35.787931 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:35.787876 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:35.804630 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:35.804436 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:35.804807 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:35.804750 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:36.058000 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:36.057883 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" podStartSLOduration=6.927161357 podStartE2EDuration="23.057870342s" totalImagesPullingTime="16.130708985s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:36.057556831 +0000 UTC m=+24.013763977" watchObservedRunningTime="2026-10-30 19:15:36.057870342 +0000 UTC m=+24.014077493" Oct 30 19:15:36.607670 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:36.607635 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:36.607670 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:36.607662 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:36.607918 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:36.607753 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:36.607918 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:36.607776 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7gj92" podUID="a619b2e9-c604-4179-aa69-aa822f209f0c" Oct 30 19:15:36.607918 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:36.607841 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vks7" podUID="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" Oct 30 19:15:36.607918 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:36.607890 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ql75b" podUID="3d1a766f-5256-4044-98d5-f551f811e684" Oct 30 19:15:36.788355 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:36.788316 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:15:36.865200 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:36.865117 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:15:36.865363 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:36.865258 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:15:36.865662 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:36.865638 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:15:37.790905 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:37.790882 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:15:38.309572 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:38.309540 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6vks7"] Oct 30 19:15:38.309767 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:38.309637 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:38.309767 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:38.309727 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vks7" podUID="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" Oct 30 19:15:38.313701 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:38.313677 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ql75b"] Oct 30 19:15:38.313829 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:38.313759 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:38.313871 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:38.313842 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ql75b" podUID="3d1a766f-5256-4044-98d5-f551f811e684" Oct 30 19:15:38.359120 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:38.359083 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7gj92"] Oct 30 19:15:38.359305 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:38.359196 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:38.359360 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:38.359306 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7gj92" podUID="a619b2e9-c604-4179-aa69-aa822f209f0c" Oct 30 19:15:39.607668 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:39.607631 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:39.608319 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:39.607641 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:39.608319 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:39.607750 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7gj92" podUID="a619b2e9-c604-4179-aa69-aa822f209f0c" Oct 30 19:15:39.608319 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:39.607649 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:39.608319 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:39.607836 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ql75b" podUID="3d1a766f-5256-4044-98d5-f551f811e684" Oct 30 19:15:39.608319 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:39.607875 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vks7" podUID="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" Oct 30 19:15:40.356432 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:40.356392 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:40.356699 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:40.356679 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:15:40.367648 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:40.367606 2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" podUID="a62fb314-b483-4dcd-8170-a818757da247" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 19:15:40.376042 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:40.376006 2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" podUID="a62fb314-b483-4dcd-8170-a818757da247" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 19:15:41.607534 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:41.607507 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:41.608010 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:41.607590 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:41.608010 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:41.607600 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ql75b" podUID="3d1a766f-5256-4044-98d5-f551f811e684" Oct 30 19:15:41.608010 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:41.607678 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7gj92" podUID="a619b2e9-c604-4179-aa69-aa822f209f0c" Oct 30 19:15:41.608010 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:41.607688 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:41.608010 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:41.607787 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vks7" podUID="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" Oct 30 19:15:43.369875 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:43.369671 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:43.608467 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:43.608444 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:43.608620 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:43.608598 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:43.608685 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:43.608612 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vks7" podUID="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" Oct 30 19:15:43.608685 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:43.608631 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:43.608759 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:43.608693 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ql75b" podUID="3d1a766f-5256-4044-98d5-f551f811e684" Oct 30 19:15:43.608799 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:43.608759 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7gj92" podUID="a619b2e9-c604-4179-aa69-aa822f209f0c" Oct 30 19:15:43.753733 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:43.753651 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:43.837352 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:43.837325 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeReady" Oct 30 19:15:43.837518 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:43.837448 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 19:15:43.959801 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:43.959767 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rp9mn"] Oct 30 19:15:43.962872 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:43.962848 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rp9mn" Oct 30 19:15:44.006030 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.005957 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4j2jg"] Oct 30 19:15:44.008784 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.008769 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:44.016842 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.016805 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:15:44.017022 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.016953 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:15:44.017086 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.017079 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-qjhd5\"" Oct 30 19:15:44.017158 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.017141 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:15:44.025667 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.025645 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rp9mn"] Oct 30 19:15:44.030130 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.030106 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4j2jg"] Oct 30 19:15:44.103890 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.103854 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/241808e4-15ac-452d-9139-85411bc375cd-cert\") pod \"ingress-canary-rp9mn\" (UID: \"241808e4-15ac-452d-9139-85411bc375cd\") " pod="openshift-ingress-canary/ingress-canary-rp9mn" Oct 30 19:15:44.103890 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.103890 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzd7b\" (UniqueName: \"kubernetes.io/projected/241808e4-15ac-452d-9139-85411bc375cd-kube-api-access-qzd7b\") pod \"ingress-canary-rp9mn\" (UID: \"241808e4-15ac-452d-9139-85411bc375cd\") " pod="openshift-ingress-canary/ingress-canary-rp9mn" Oct 30 19:15:44.104133 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.103910 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6xm9\" (UniqueName: \"kubernetes.io/projected/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-kube-api-access-c6xm9\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:44.104133 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.104049 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:44.104133 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.104101 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-config-volume\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:44.104133 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.104120 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-tmp-dir\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:44.119328 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.119280 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:15:44.119439 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.119394 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hk6fn\"" Oct 30 19:15:44.121428 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.121410 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:15:44.205383 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.205346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-config-volume\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:44.205383 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.205380 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-tmp-dir\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:44.205619 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.205423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/241808e4-15ac-452d-9139-85411bc375cd-cert\") pod \"ingress-canary-rp9mn\" (UID: \"241808e4-15ac-452d-9139-85411bc375cd\") " pod="openshift-ingress-canary/ingress-canary-rp9mn" Oct 30 19:15:44.205619 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.205440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzd7b\" (UniqueName: \"kubernetes.io/projected/241808e4-15ac-452d-9139-85411bc375cd-kube-api-access-qzd7b\") pod \"ingress-canary-rp9mn\" (UID: \"241808e4-15ac-452d-9139-85411bc375cd\") " pod="openshift-ingress-canary/ingress-canary-rp9mn" Oct 30 19:15:44.205619 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.205586 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6xm9\" (UniqueName: \"kubernetes.io/projected/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-kube-api-access-c6xm9\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:44.205765 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.205630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:44.205765 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:44.205742 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:15:44.205850 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:44.205809 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls podName:7dcbf677-ca92-4306-bc45-ab9954ed7ac2 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:44.705788416 +0000 UTC m=+32.661995558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls") pod "dns-default-4j2jg" (UID: "7dcbf677-ca92-4306-bc45-ab9954ed7ac2") : secret "dns-default-metrics-tls" not found Oct 30 19:15:44.205850 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.205840 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-tmp-dir\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:44.206101 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.206083 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-config-volume\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:44.209692 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.209673 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/241808e4-15ac-452d-9139-85411bc375cd-cert\") pod \"ingress-canary-rp9mn\" (UID: \"241808e4-15ac-452d-9139-85411bc375cd\") " pod="openshift-ingress-canary/ingress-canary-rp9mn" Oct 30 19:15:44.263107 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.263027 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6xm9\" (UniqueName: \"kubernetes.io/projected/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-kube-api-access-c6xm9\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:44.263262 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.263246 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzd7b\" (UniqueName: \"kubernetes.io/projected/241808e4-15ac-452d-9139-85411bc375cd-kube-api-access-qzd7b\") pod \"ingress-canary-rp9mn\" (UID: \"241808e4-15ac-452d-9139-85411bc375cd\") " pod="openshift-ingress-canary/ingress-canary-rp9mn" Oct 30 19:15:44.274077 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.274051 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rp9mn" Oct 30 19:15:44.660228 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.660197 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rp9mn"] Oct 30 19:15:44.663731 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:44.663701 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241808e4_15ac_452d_9139_85411bc375cd.slice/crio-8df58a0005cdf247c07c538f0ecb2e97ca6d81abff60714e004d080e9c579a9e WatchSource:0}: Error finding container 8df58a0005cdf247c07c538f0ecb2e97ca6d81abff60714e004d080e9c579a9e: Status 404 returned error can't find the container with id 8df58a0005cdf247c07c538f0ecb2e97ca6d81abff60714e004d080e9c579a9e Oct 30 19:15:44.709636 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.709602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:44.709768 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:44.709742 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:15:44.709817 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:44.709805 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls podName:7dcbf677-ca92-4306-bc45-ab9954ed7ac2 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:45.709789781 +0000 UTC m=+33.665996913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls") pod "dns-default-4j2jg" (UID: "7dcbf677-ca92-4306-bc45-ab9954ed7ac2") : secret "dns-default-metrics-tls" not found Oct 30 19:15:44.805537 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:44.805506 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rp9mn" event={"ID":"241808e4-15ac-452d-9139-85411bc375cd","Type":"ContainerStarted","Data":"8df58a0005cdf247c07c538f0ecb2e97ca6d81abff60714e004d080e9c579a9e"} Oct 30 19:15:45.608091 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:45.608045 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:45.608091 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:45.608070 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:45.608328 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:45.608244 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:45.660289 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:45.660253 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:15:45.660289 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:45.660265 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:15:45.709836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:45.709797 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-px7h6\"" Oct 30 19:15:45.709836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:45.709815 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:15:45.709836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:45.709830 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:15:45.709836 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:45.709815 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jlfgq\"" Oct 30 19:15:45.710097 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:45.709815 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gt8zl\"" Oct 30 19:15:45.718259 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:45.718234 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:45.718385 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:45.718371 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:15:45.718436 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:45.718425 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls podName:7dcbf677-ca92-4306-bc45-ab9954ed7ac2 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:47.71841054 +0000 UTC m=+35.674617669 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls") pod "dns-default-4j2jg" (UID: "7dcbf677-ca92-4306-bc45-ab9954ed7ac2") : secret "dns-default-metrics-tls" not found Oct 30 19:15:46.322279 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:46.322240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:46.324707 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:46.324679 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:46.423168 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:46.423130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:46.423331 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:46.423279 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:15:46.423389 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:46.423343 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs podName:a619b2e9-c604-4179-aa69-aa822f209f0c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:18.423325815 +0000 UTC m=+66.379532959 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs") pod "network-metrics-daemon-7gj92" (UID: "a619b2e9-c604-4179-aa69-aa822f209f0c") : secret "metrics-daemon-secret" not found Oct 30 19:15:46.523612 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:46.523571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrg7\" (UniqueName: \"kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7\") pod \"network-check-target-ql75b\" (UID: \"3d1a766f-5256-4044-98d5-f551f811e684\") " pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:46.526410 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:46.526384 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrg7\" (UniqueName: \"kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7\") pod \"network-check-target-ql75b\" (UID: \"3d1a766f-5256-4044-98d5-f551f811e684\") " pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:46.568281 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:46.568245 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:46.587640 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:46.587615 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:46.632490 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:46.632446 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:46.908390 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:46.908309 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:46.974553 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:46.974523 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6vks7"] Oct 30 19:15:46.978162 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:46.978125 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded1dc05f_2a13_4dd2_9d20_f5e96f17099f.slice/crio-636897ff0a85c7e3e55a3bb1f8f79ae33a4f9e39d8c35e23151a523e7b1bf2d4 WatchSource:0}: Error finding container 636897ff0a85c7e3e55a3bb1f8f79ae33a4f9e39d8c35e23151a523e7b1bf2d4: Status 404 returned error can't find the container with id 636897ff0a85c7e3e55a3bb1f8f79ae33a4f9e39d8c35e23151a523e7b1bf2d4 Oct 30 19:15:47.023518 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:47.023484 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ql75b"] Oct 30 19:15:47.027275 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:47.027253 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1a766f_5256_4044_98d5_f551f811e684.slice/crio-6c8f10c4848ba9278ffe38fe6eeebdf1bcf4f34fa889e63eb568816b6b543af8 WatchSource:0}: Error finding container 6c8f10c4848ba9278ffe38fe6eeebdf1bcf4f34fa889e63eb568816b6b543af8: Status 404 returned error can't find the container with id 6c8f10c4848ba9278ffe38fe6eeebdf1bcf4f34fa889e63eb568816b6b543af8 Oct 30 19:15:47.733087 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:47.733049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:47.733253 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:47.733230 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:15:47.733304 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:47.733294 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls podName:7dcbf677-ca92-4306-bc45-ab9954ed7ac2 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:51.733279618 +0000 UTC m=+39.689486760 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls") pod "dns-default-4j2jg" (UID: "7dcbf677-ca92-4306-bc45-ab9954ed7ac2") : secret "dns-default-metrics-tls" not found Oct 30 19:15:47.812645 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:47.812602 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ql75b" event={"ID":"3d1a766f-5256-4044-98d5-f551f811e684","Type":"ContainerStarted","Data":"6c8f10c4848ba9278ffe38fe6eeebdf1bcf4f34fa889e63eb568816b6b543af8"} Oct 30 19:15:47.813755 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:47.813719 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6vks7" event={"ID":"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f","Type":"ContainerStarted","Data":"636897ff0a85c7e3e55a3bb1f8f79ae33a4f9e39d8c35e23151a523e7b1bf2d4"} Oct 30 19:15:48.818609 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:48.818564 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rp9mn" event={"ID":"241808e4-15ac-452d-9139-85411bc375cd","Type":"ContainerStarted","Data":"11fe0ee0c3737545268eb88c2aa917d2841884b8c86b567bf7bc56b36fcf948f"} Oct 30 19:15:48.932478 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:48.932416 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rp9mn" podStartSLOduration=2.756953502 podStartE2EDuration="5.93240066s" totalImagesPullingTime="3.175447158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:48.932004799 +0000 UTC m=+36.888211944" watchObservedRunningTime="2026-10-30 19:15:48.93240066 +0000 UTC m=+36.888607806" Oct 30 19:15:49.305022 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.304916 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:49.499376 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.499339 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fs446"] Oct 30 19:15:49.531638 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.531607 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fs446" Oct 30 19:15:49.562256 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.562222 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fs446"] Oct 30 19:15:49.577101 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.577061 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hjh7f"] Oct 30 19:15:49.601698 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.601645 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:49.603351 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.603319 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:49.625754 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.625718 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 19:15:49.625754 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.625729 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 19:15:49.626041 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.625778 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4h2hd\"" Oct 30 19:15:49.646299 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.646264 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq9wm\" (UniqueName: \"kubernetes.io/projected/d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0-kube-api-access-jq9wm\") pod \"migrator-66df5d496b-fs446\" (UID: \"d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fs446" Oct 30 19:15:49.672262 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.672225 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hjh7f"] Oct 30 19:15:49.684650 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.684619 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:15:49.684824 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.684687 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:15:49.684824 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.684730 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:15:49.684824 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.684815 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:15:49.684956 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.684862 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mnhjf\"" Oct 30 19:15:49.747724 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.747686 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-data-volume\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:49.747911 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.747738 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw5gw\" (UniqueName: \"kubernetes.io/projected/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-api-access-cw5gw\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:49.747911 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.747799 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:49.747911 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.747881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq9wm\" (UniqueName: \"kubernetes.io/projected/d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0-kube-api-access-jq9wm\") pod \"migrator-66df5d496b-fs446\" (UID: \"d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fs446" Oct 30 19:15:49.748104 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.747967 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-crio-socket\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:49.748104 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.748019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:49.804894 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.804855 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq9wm\" (UniqueName: \"kubernetes.io/projected/d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0-kube-api-access-jq9wm\") pod \"migrator-66df5d496b-fs446\" (UID: \"d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fs446" Oct 30 19:15:49.828837 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.828760 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-hjklm-r9jn7"] Oct 30 19:15:49.844061 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.844030 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:49.849160 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.849126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-data-volume\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:49.849321 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.849167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw5gw\" (UniqueName: \"kubernetes.io/projected/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-api-access-cw5gw\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:49.849321 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.849221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:49.849321 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.849276 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-crio-socket\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:49.849321 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.849296 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:49.849449 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:49.849375 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:15:49.849449 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:49.849448 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-rbac-proxy-cm podName:69a39e1d-7ab3-40a0-9f5d-ae469a5d326c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:50.34943428 +0000 UTC m=+38.305641408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hjh7f" (UID: "69a39e1d-7ab3-40a0-9f5d-ae469a5d326c") : configmap "kube-rbac-proxy" not found Oct 30 19:15:49.849775 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.849756 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-data-volume\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:49.850163 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:49.850144 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:15:49.850238 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:49.850202 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls podName:69a39e1d-7ab3-40a0-9f5d-ae469a5d326c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:50.350185068 +0000 UTC m=+38.306392197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjh7f" (UID: "69a39e1d-7ab3-40a0-9f5d-ae469a5d326c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:15:49.850414 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.850394 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-crio-socket\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:49.851137 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.851108 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:49.857966 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.857943 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fs446" Oct 30 19:15:49.892174 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.892101 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hjklm-r9jn7"] Oct 30 19:15:49.896680 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.896411 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 19:15:49.897238 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.897217 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4qn9x\"" Oct 30 19:15:49.947242 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.947079 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 19:15:49.950001 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.949968 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-service-ca-bundle\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:49.950125 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.950103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/44f7fda2-c7d3-470c-9080-1f931fac9baa-archives-path\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:49.950384 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.950368 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-trusted-ca-bundle\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:49.950439 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.950403 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh4p5\" (UniqueName: \"kubernetes.io/projected/44f7fda2-c7d3-470c-9080-1f931fac9baa-kube-api-access-jh4p5\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:49.950905 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:49.950883 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw5gw\" (UniqueName: \"kubernetes.io/projected/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-api-access-cw5gw\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:50.051045 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.051006 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-service-ca-bundle\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:50.051214 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.051068 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/44f7fda2-c7d3-470c-9080-1f931fac9baa-archives-path\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:50.051214 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.051113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-trusted-ca-bundle\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:50.051214 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.051145 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh4p5\" (UniqueName: \"kubernetes.io/projected/44f7fda2-c7d3-470c-9080-1f931fac9baa-kube-api-access-jh4p5\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:50.051586 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.051553 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-service-ca-bundle\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:50.051852 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.051832 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/44f7fda2-c7d3-470c-9080-1f931fac9baa-archives-path\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:50.052255 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.052236 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-trusted-ca-bundle\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:50.073711 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.073677 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:50.108119 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.108056 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh4p5\" (UniqueName: \"kubernetes.io/projected/44f7fda2-c7d3-470c-9080-1f931fac9baa-kube-api-access-jh4p5\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:50.173928 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.173891 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:50.276248 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.276214 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fs446"] Oct 30 19:15:50.279223 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:50.279189 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f1e0ce_e2d3_4838_bd81_7138a77b5aa0.slice/crio-b2538555ee0daaad4942a4aeb2b16e83f115d481c85cae536b5706bff296827f WatchSource:0}: Error finding container b2538555ee0daaad4942a4aeb2b16e83f115d481c85cae536b5706bff296827f: Status 404 returned error can't find the container with id b2538555ee0daaad4942a4aeb2b16e83f115d481c85cae536b5706bff296827f Oct 30 19:15:50.304815 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.304664 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:50.356876 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.356840 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:50.357093 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.356920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:50.357093 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:50.357024 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:15:50.357093 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:50.357093 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-rbac-proxy-cm podName:69a39e1d-7ab3-40a0-9f5d-ae469a5d326c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:51.357074945 +0000 UTC m=+39.313282151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hjh7f" (UID: "69a39e1d-7ab3-40a0-9f5d-ae469a5d326c") : configmap "kube-rbac-proxy" not found Oct 30 19:15:50.357522 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:50.357498 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:15:50.357713 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:50.357559 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls podName:69a39e1d-7ab3-40a0-9f5d-ae469a5d326c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:51.357541023 +0000 UTC m=+39.313748155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjh7f" (UID: "69a39e1d-7ab3-40a0-9f5d-ae469a5d326c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:15:50.543548 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.543450 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:50.579236 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.579196 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hjklm-r9jn7"] Oct 30 19:15:50.583477 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:50.583447 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f7fda2_c7d3_470c_9080_1f931fac9baa.slice/crio-e6d43012785906e18c9802e1072e7f84a90b3dd1c4ff63b59cb094da8141f808 WatchSource:0}: Error finding container e6d43012785906e18c9802e1072e7f84a90b3dd1c4ff63b59cb094da8141f808: Status 404 returned error can't find the container with id e6d43012785906e18c9802e1072e7f84a90b3dd1c4ff63b59cb094da8141f808 Oct 30 19:15:50.765363 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.765323 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:50.825875 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.825834 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hjklm-r9jn7" event={"ID":"44f7fda2-c7d3-470c-9080-1f931fac9baa","Type":"ContainerStarted","Data":"e6d43012785906e18c9802e1072e7f84a90b3dd1c4ff63b59cb094da8141f808"} Oct 30 19:15:50.827147 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:50.827113 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fs446" event={"ID":"d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0","Type":"ContainerStarted","Data":"b2538555ee0daaad4942a4aeb2b16e83f115d481c85cae536b5706bff296827f"} Oct 30 19:15:51.301421 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.301338 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-m92f6"] Oct 30 19:15:51.316136 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.316097 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:51.361967 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.361943 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-m92f6"] Oct 30 19:15:51.365317 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.365265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:51.366139 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.365798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:51.366139 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:51.365944 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:15:51.366139 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:51.366025 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls podName:69a39e1d-7ab3-40a0-9f5d-ae469a5d326c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:53.366004641 +0000 UTC m=+41.322211785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjh7f" (UID: "69a39e1d-7ab3-40a0-9f5d-ae469a5d326c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:15:51.366139 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:51.366077 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:15:51.366139 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:51.366108 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-rbac-proxy-cm podName:69a39e1d-7ab3-40a0-9f5d-ae469a5d326c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:53.366097754 +0000 UTC m=+41.322304896 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hjh7f" (UID: "69a39e1d-7ab3-40a0-9f5d-ae469a5d326c") : configmap "kube-rbac-proxy" not found Oct 30 19:15:51.369840 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.369816 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 19:15:51.369967 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.369946 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 19:15:51.370938 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.370915 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 19:15:51.371063 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.371041 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6wdgf\"" Oct 30 19:15:51.375550 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.371272 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 19:15:51.466288 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.466255 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-signing-cabundle\") pod \"service-ca-5fffc97bdf-m92f6\" (UID: \"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:51.466449 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.466323 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-signing-key\") pod \"service-ca-5fffc97bdf-m92f6\" (UID: \"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:51.466521 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.466495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8gb9\" (UniqueName: \"kubernetes.io/projected/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-kube-api-access-k8gb9\") pod \"service-ca-5fffc97bdf-m92f6\" (UID: \"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:51.567565 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.567531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8gb9\" (UniqueName: \"kubernetes.io/projected/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-kube-api-access-k8gb9\") pod \"service-ca-5fffc97bdf-m92f6\" (UID: \"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:51.567744 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.567686 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-signing-cabundle\") pod \"service-ca-5fffc97bdf-m92f6\" (UID: \"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:51.567744 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.567722 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-signing-key\") pod \"service-ca-5fffc97bdf-m92f6\" (UID: \"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:51.568600 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.568571 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-signing-cabundle\") pod \"service-ca-5fffc97bdf-m92f6\" (UID: \"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:51.570675 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.570649 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-signing-key\") pod \"service-ca-5fffc97bdf-m92f6\" (UID: \"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:51.625158 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.625101 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8gb9\" (UniqueName: \"kubernetes.io/projected/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-kube-api-access-k8gb9\") pod \"service-ca-5fffc97bdf-m92f6\" (UID: \"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:51.650884 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.650844 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:51.667144 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.667013 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:51.770100 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.769507 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:51.770100 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:51.769686 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:15:51.770100 ip-10-0-141-5 kubenswrapper[2577]: E1030 19:15:51.769753 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls podName:7dcbf677-ca92-4306-bc45-ab9954ed7ac2 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:59.769733536 +0000 UTC m=+47.725940668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls") pod "dns-default-4j2jg" (UID: "7dcbf677-ca92-4306-bc45-ab9954ed7ac2") : secret "dns-default-metrics-tls" not found Oct 30 19:15:51.832937 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.832436 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ql75b" event={"ID":"3d1a766f-5256-4044-98d5-f551f811e684","Type":"ContainerStarted","Data":"3f894640f7e551916c856d66f822dc249ca8fb814cf72d7abbafb0beac68d406"} Oct 30 19:15:51.832937 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.832657 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:51.953057 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:51.952957 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:52.060165 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:52.059277 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ql75b" podStartSLOduration=33.764179774 podStartE2EDuration="38.059256065s" totalImagesPullingTime="4.295076291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:51.948452845 +0000 UTC m=+39.904659994" watchObservedRunningTime="2026-10-30 19:15:52.059256065 +0000 UTC m=+40.015463217" Oct 30 19:15:52.060165 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:52.060089 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-m92f6"] Oct 30 19:15:52.064159 ip-10-0-141-5 kubenswrapper[2577]: W1030 19:15:52.064130 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d921aae_0b5b_4f3f_9ad5_33fcacb7213b.slice/crio-23d24245e8b069e7f3cf92d29fbcd4bf6ac16b18cd49bd593e33b0916615cff3 WatchSource:0}: Error finding container 23d24245e8b069e7f3cf92d29fbcd4bf6ac16b18cd49bd593e33b0916615cff3: Status 404 returned error can't find the container with id 23d24245e8b069e7f3cf92d29fbcd4bf6ac16b18cd49bd593e33b0916615cff3 Oct 30 19:15:52.487423 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:52.487392 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:52.712752 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:52.712668 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:52.836529 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:52.836492 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fs446" event={"ID":"d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0","Type":"ContainerStarted","Data":"834565cd3a42cb0ee43e20398c69b2946b5a1ef2617348d7848198ef50ada5a8"} Oct 30 19:15:52.838365 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:52.838015 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6vks7" event={"ID":"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f","Type":"ContainerStarted","Data":"1268c0c52cd83f7558086182da42bc6a4314438e0935f4fd60e916a04983c3c4"} Oct 30 19:15:52.839135 ip-10-0-141-5 kubenswrapper[2577]: I1030 19:15:52.839111 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" event={"ID":"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b","Type":"ContainerStarted","Data":"23d24245e8b069e7f3cf92d29fbcd4bf6ac16b18cd49bd593e33b0916615cff3"} Oct 30 19:15:52.995220 ip-10-0-141-5 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 19:15:53.012429 ip-10-0-141-5 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 19:15:53.012606 ip-10-0-141-5 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 19:15:53.012656 ip-10-0-141-5 systemd[1]: kubelet.service: Consumed 2.488s CPU time. Oct 30 19:15:53.024683 ip-10-0-141-5 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:15:53.116822 ip-10-0-141-5 kubenswrapper[4433]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:15:53.116822 ip-10-0-141-5 kubenswrapper[4433]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:15:53.116822 ip-10-0-141-5 kubenswrapper[4433]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:15:53.116822 ip-10-0-141-5 kubenswrapper[4433]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:15:53.119321 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119229 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:53.119321 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119320 4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:53.119321 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119328 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119332 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119337 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119343 4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119346 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119349 4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119352 4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119355 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119358 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119362 4433 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119366 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119370 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119375 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119379 4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119383 4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119387 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119390 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119392 4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119396 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119400 4433 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:53.119505 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119404 4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119409 4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119414 4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119419 4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119424 4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119429 4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119440 4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119443 4433 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119446 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119449 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119451 4433 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119454 4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119457 4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119459 4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119462 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119466 4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119470 4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119474 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119478 4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119482 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:53.120131 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119486 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119490 4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119494 4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119498 4433 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119502 4433 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119506 4433 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119510 4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119514 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119517 4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119519 4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119522 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119524 4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119527 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119530 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119533 4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119535 4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119538 4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119540 4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119543 4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119553 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:53.120827 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119556 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119561 4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119564 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119569 4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119573 4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119579 4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119583 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119587 4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119591 4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119596 4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119604 4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119609 4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119613 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119618 4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119622 4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119625 4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119627 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119630 4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119632 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:53.121487 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119635 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119637 4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119640 4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119642 4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119645 4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119648 4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119650 4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119653 4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119655 4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119658 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119660 4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119663 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119665 4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119677 4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119681 4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119684 4433 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119687 4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119690 4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119693 4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119695 4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:53.122220 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119698 4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119700 4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119705 4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119709 4433 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.119713 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120791 4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120801 4433 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120804 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120807 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120809 4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120812 4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120814 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120817 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120820 4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120823 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120825 4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120828 4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120830 4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120833 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:53.122909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120836 4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120838 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120841 4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120844 4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120847 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120850 4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120853 4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120856 4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120858 4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120863 4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120867 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120872 4433 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120877 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120881 4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120886 4433 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120890 4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120893 4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120896 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120899 4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120902 4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:53.123701 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120904 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120907 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120909 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120912 4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120914 4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120917 4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120920 4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120922 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120925 4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120927 4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120930 4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120933 4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120935 4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120937 4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120940 4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120942 4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120944 4433 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120947 4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120949 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120951 4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:53.124523 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120954 4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120957 4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120960 4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120963 4433 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120966 4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120969 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120973 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120990 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120994 4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.120998 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121015 4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121019 4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121023 4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121027 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121031 4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121035 4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121039 4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121043 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121046 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121050 4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:53.125357 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121054 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121059 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121064 4433 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121069 4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121072 4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121076 4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121080 4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121084 4433 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121089 4433 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121094 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121098 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121102 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121107 4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121111 4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121116 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121120 4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121127 4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121133 4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121137 4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121140 4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:53.126188 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121143 4433 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121147 4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121149 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121152 4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121154 4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121157 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121159 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121162 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121164 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121167 4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121170 4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.121174 4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122807 4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122819 4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122825 4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122831 4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122835 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122840 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122851 4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:53.126933 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122856 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122859 4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122862 4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122865 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122868 4433 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122871 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122875 4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122878 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122881 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122883 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122886 4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122889 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122894 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122897 4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122899 4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122902 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122905 4433 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122908 4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122910 4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122914 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:53.127640 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122916 4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122919 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122922 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122925 4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122927 4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122933 4433 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122938 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122942 4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122946 4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122950 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122955 4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122960 4433 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122964 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122969 4433 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122973 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122990 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.122995 4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123002 4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123005 4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123009 4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:53.128381 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123012 4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123014 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123017 4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123019 4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123022 4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123025 4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123027 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123030 4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123033 4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123035 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123040 4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123043 4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123046 4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123052 4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123055 4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123059 4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123062 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123065 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123067 4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:53.129182 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123070 4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123072 4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123076 4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123083 4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123086 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123089 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123092 4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123095 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123098 4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123104 4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123109 4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123114 4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123118 4433 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123122 4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123125 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123128 4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123133 4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123136 4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123138 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:53.129886 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123141 4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123144 4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123146 4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123149 4433 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123152 4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123154 4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123157 4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123160 4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123162 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123167 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123170 4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123173 4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123175 4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123178 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123180 4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123183 4433 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123185 4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123188 4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123191 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123201 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:53.130638 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.123205 4433 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123351 4433 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123367 4433 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123377 4433 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123384 4433 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123391 4433 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123397 4433 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123404 4433 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123410 4433 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123416 4433 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123419 4433 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123423 4433 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123427 4433 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123430 4433 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123433 4433 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123436 4433 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123440 4433 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123442 4433 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123448 4433 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123452 4433 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123455 4433 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123458 4433 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123461 4433 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123465 4433 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:15:53.131383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123469 4433 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123472 4433 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123478 4433 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123482 4433 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123485 4433 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123490 4433 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123494 4433 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123502 4433 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123506 4433 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123511 4433 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123514 4433 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123519 4433 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123522 4433 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123526 4433 flags.go:64] FLAG: --enable-server="true" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123529 4433 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123534 4433 flags.go:64] FLAG: --event-burst="100" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123537 4433 flags.go:64] FLAG: --event-qps="50" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123540 4433 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123544 4433 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123549 4433 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123553 4433 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123556 4433 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123559 4433 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123562 4433 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123565 4433 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:15:53.132355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123568 4433 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123571 4433 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123574 4433 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123581 4433 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123586 4433 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123591 4433 flags.go:64] FLAG: --feature-gates="" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123598 4433 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123603 4433 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123609 4433 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123614 4433 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123620 4433 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123629 4433 flags.go:64] FLAG: --help="false" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123634 4433 flags.go:64] FLAG: --hostname-override="ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123640 4433 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123645 4433 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123648 4433 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123651 4433 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123655 4433 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123658 4433 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123661 4433 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123698 4433 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123722 4433 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123728 4433 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:15:53.133342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123734 4433 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123741 4433 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123746 4433 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123751 4433 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123756 4433 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123759 4433 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123765 4433 flags.go:64] FLAG: --lock-file="" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123768 4433 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123771 4433 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123774 4433 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123780 4433 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123783 4433 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123787 4433 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123790 4433 flags.go:64] FLAG: --logging-format="text" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123793 4433 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123799 4433 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123802 4433 flags.go:64] FLAG: --manifest-url="" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123805 4433 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123810 4433 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123813 4433 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123817 4433 flags.go:64] FLAG: --max-pods="110" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123821 4433 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123824 4433 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123834 4433 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123839 4433 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:15:53.134243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123844 4433 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123849 4433 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123855 4433 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123868 4433 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123873 4433 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123882 4433 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123886 4433 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123889 4433 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123892 4433 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123895 4433 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123898 4433 flags.go:64] FLAG: --port="10250" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123902 4433 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123905 4433 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-09695772b70aab656" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123910 4433 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123913 4433 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123917 4433 flags.go:64] FLAG: --register-node="true" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123919 4433 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123923 4433 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123926 4433 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123931 4433 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123934 4433 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123938 4433 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123944 4433 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123947 4433 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123951 4433 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123954 4433 flags.go:64] FLAG: --runonce="false" Oct 30 19:15:53.135158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123957 4433 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123960 4433 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123964 4433 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123967 4433 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123973 4433 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123990 4433 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123995 4433 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.123998 4433 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124120 4433 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124127 4433 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124131 4433 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124138 4433 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124143 4433 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124146 4433 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124149 4433 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124160 4433 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124165 4433 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124169 4433 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124175 4433 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124178 4433 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124181 4433 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124184 4433 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124187 4433 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124190 4433 flags.go:64] FLAG: --v="2" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124195 4433 flags.go:64] FLAG: --version="false" Oct 30 19:15:53.136043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124203 4433 flags.go:64] FLAG: --vmodule="" Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124221 4433 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124225 4433 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124382 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124387 4433 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124391 4433 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124394 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124397 4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124400 4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124403 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124405 4433 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124410 4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124412 4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124416 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124419 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124421 4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124424 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124427 4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124431 4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124435 4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:53.136996 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124439 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124443 4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124446 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124449 4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124453 4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124458 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124462 4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124465 4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124468 4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124470 4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124473 4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124475 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124479 4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124484 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124487 4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124491 4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124495 4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124498 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124501 4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:53.137668 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124504 4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124508 4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124512 4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124517 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124523 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124526 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124529 4433 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124533 4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124537 4433 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124540 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124544 4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124546 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124550 4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124553 4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124556 4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124565 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124569 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124571 4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124574 4433 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124576 4433 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:53.138331 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124579 4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124581 4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124585 4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124589 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124594 4433 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124600 4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124603 4433 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124606 4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124610 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124614 4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124618 4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124622 4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124625 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124627 4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124630 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124633 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124636 4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124638 4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124641 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124644 4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:53.139022 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124646 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124649 4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124651 4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124653 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124656 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124658 4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124661 4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124664 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124668 4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124672 4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124676 4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124681 4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124684 4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124686 4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124689 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124692 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124695 4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124699 4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124702 4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:53.139696 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124709 4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:53.140359 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124714 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:53.140359 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124718 4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:53.140359 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124723 4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:53.140359 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124727 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:53.140359 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124731 4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:53.140359 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124736 4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:53.140359 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124740 4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:53.140359 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124744 4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:53.140359 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124750 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:53.140359 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124754 4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:53.140359 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.124794 4433 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:15:53.140359 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124839 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:53.140359 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124844 4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:53.140359 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124847 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:53.140359 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124851 4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124856 4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124860 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124865 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124869 4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124872 4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124875 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124877 4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124880 4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124882 4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124885 4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124888 4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124891 4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124893 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124896 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124899 4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124904 4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124906 4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124910 4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:53.140864 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124914 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124918 4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124921 4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124923 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124926 4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124929 4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124931 4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124934 4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124936 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124939 4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124941 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124944 4433 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124946 4433 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124949 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124952 4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124954 4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124957 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124959 4433 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124962 4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124965 4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:53.141535 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124969 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124975 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124996 4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.124999 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125002 4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125004 4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125007 4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125009 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125013 4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125017 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125020 4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125023 4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125025 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125027 4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125030 4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125033 4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125037 4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125040 4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125042 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:53.142377 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125045 4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125047 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125050 4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125052 4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125055 4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125057 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125060 4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125063 4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125065 4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125068 4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125070 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125073 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125075 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125080 4433 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125086 4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125092 4433 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125096 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125099 4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125102 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:53.143185 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125104 4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125107 4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125109 4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125114 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125117 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125120 4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125122 4433 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125125 4433 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125127 4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125130 4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125133 4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125136 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125139 4433 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125141 4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125143 4433 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125146 4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125148 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125151 4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125153 4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125157 4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125161 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:53.143878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125166 4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125170 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125173 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125176 4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.125178 4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.129668 4433 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.129683 4433 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.129706 4433 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.129712 4433 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129792 4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129800 4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129805 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129810 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129814 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129822 4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129826 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129830 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129834 4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129838 4433 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129842 4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:53.144648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129845 4433 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129847 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129850 4433 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129854 4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129859 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129864 4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129868 4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129873 4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129877 4433 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129881 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129885 4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129889 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129892 4433 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129895 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129898 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129901 4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129905 4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129907 4433 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129910 4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:53.145312 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129913 4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129917 4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129924 4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129929 4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129934 4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129939 4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129943 4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129946 4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129951 4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129955 4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129960 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129964 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129966 4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129969 4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129972 4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129975 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129992 4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.129996 4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130001 4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:53.146001 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130006 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130011 4433 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130015 4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130018 4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130022 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130024 4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130027 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130031 4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130035 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130038 4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130041 4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130044 4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130047 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130051 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130055 4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130060 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130064 4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130069 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130072 4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:53.146552 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130075 4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130080 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130084 4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130090 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130094 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130098 4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130101 4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130104 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130108 4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130112 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130117 4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130121 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130125 4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130129 4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130134 4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130138 4433 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130142 4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130147 4433 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130149 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:53.147227 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130152 4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130155 4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130157 4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130160 4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130163 4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130165 4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130168 4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130171 4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130175 4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130179 4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130183 4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130187 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130190 4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130194 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130198 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130203 4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130208 4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130212 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:53.147878 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130217 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:53.148614 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.130260 4433 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:15:53.148614 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130428 4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:53.148614 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130436 4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:53.148614 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130441 4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:53.148614 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130448 4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:53.148614 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130453 4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:53.148614 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130457 4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:53.148614 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130462 4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:53.148614 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130467 4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:53.148614 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130471 4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:53.148614 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130477 4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:53.148614 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130481 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:53.148614 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130485 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:53.148614 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130490 4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:53.148614 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130494 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130497 4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130500 4433 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130503 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130507 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130511 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130516 4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130520 4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130523 4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130528 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130532 4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130535 4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130539 4433 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130543 4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130547 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130551 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130555 4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130558 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130561 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130564 4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:53.149124 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130568 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130572 4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130576 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130579 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130583 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130587 4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130591 4433 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130596 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130600 4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130605 4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130609 4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130613 4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130617 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130620 4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130624 4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130628 4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130632 4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130635 4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130640 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130643 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:53.149862 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130646 4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130649 4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130653 4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130658 4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130663 4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130667 4433 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130672 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130676 4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130678 4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130681 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130686 4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130690 4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130693 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130695 4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130698 4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130700 4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130703 4433 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130705 4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130708 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:53.150459 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130712 4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130716 4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130721 4433 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130725 4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130729 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130733 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130737 4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130741 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130746 4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130750 4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130753 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130755 4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130758 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130761 4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130763 4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130766 4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130768 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130770 4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130773 4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130775 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:53.151064 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130778 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:53.151568 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130782 4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:53.151568 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130786 4433 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:53.151568 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130790 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:53.151568 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130796 4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:53.151568 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130800 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:53.151568 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130808 4433 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:53.151568 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130813 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:53.151568 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130817 4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:53.151568 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130822 4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:53.151568 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130826 4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:53.151568 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130830 4433 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:53.151568 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130835 4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:53.151568 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130839 4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:53.151568 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.130872 4433 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:15:53.151568 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130906 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:53.151568 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130911 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130913 4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130916 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130918 4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130921 4433 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130923 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130926 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130928 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130931 4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130933 4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130936 4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130939 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130941 4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130944 4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130946 4433 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130949 4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130951 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130953 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130956 4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130958 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:53.152104 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130961 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130964 4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130970 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130974 4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130991 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.130996 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131000 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131005 4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131009 4433 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131012 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131015 4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131017 4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131020 4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131022 4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131025 4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131027 4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131030 4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131035 4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131039 4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131043 4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:53.152935 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131048 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131052 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131056 4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131060 4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131065 4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131069 4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131074 4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131077 4433 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131081 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131086 4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131090 4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131093 4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131100 4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131105 4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131114 4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131119 4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131123 4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131127 4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131132 4433 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:53.153558 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131136 4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131140 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131144 4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131148 4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131153 4433 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131157 4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131161 4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131165 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131169 4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131172 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131175 4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131178 4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131180 4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131183 4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131187 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131191 4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131195 4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131199 4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131203 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131207 4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:53.154039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131211 4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131215 4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131219 4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131223 4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131226 4433 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131230 4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131233 4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131237 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131241 4433 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131243 4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131248 4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131252 4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131257 4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131261 4433 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131265 4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131269 4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131273 4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131276 4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131278 4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:53.154559 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131282 4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:53.155122 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131288 4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:53.155122 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131293 4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:53.155122 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131297 4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:53.155122 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131301 4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:53.155122 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131307 4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:53.155122 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:15:53.131311 4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:53.155122 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.131377 4433 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:15:53.155122 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.132596 4433 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 19:15:53.155122 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.132661 4433 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 19:15:53.155122 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.132869 4433 server.go:1014] "Starting client certificate rotation" Oct 30 19:15:53.155122 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.133020 4433 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:15:53.155122 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.133089 4433 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:10:13 +0000 UTC" deadline="2028-04-12 03:32:47.603101525 +0000 UTC" Oct 30 19:15:53.155122 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.133129 4433 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12704h16m54.469975917s" Oct 30 19:15:53.155122 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.133437 4433 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:15:53.155122 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.134939 4433 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:15:53.155122 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.136736 4433 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:15:53.155608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.138558 4433 log.go:25] "Validated CRI v1 image API" Oct 30 19:15:53.155608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.139453 4433 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:15:53.155608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.141377 4433 fs.go:135] Filesystem UUIDs: map[2cbb70df-2a1d-4d46-895f-43f8b25f0b0e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 caf7e14a-2aa0-430a-8b3d-1bd06f7c283a:/dev/nvme0n1p3] Oct 30 19:15:53.155711 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.141396 4433 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00f7c9795076d2276b5d53c2f77ece72abdba65e4e5b47dcd39cd0350e710ff8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00f7c9795076d2276b5d53c2f77ece72abdba65e4e5b47dcd39cd0350e710ff8/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1eee746462d68b33b3b02fe2ac8ca0126457fa5e52540af045fe141565c717a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1eee746462d68b33b3b02fe2ac8ca0126457fa5e52540af045fe141565c717a5/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23d24245e8b069e7f3cf92d29fbcd4bf6ac16b18cd49bd593e33b0916615cff3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23d24245e8b069e7f3cf92d29fbcd4bf6ac16b18cd49bd593e33b0916615cff3/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/636897ff0a85c7e3e55a3bb1f8f79ae33a4f9e39d8c35e23151a523e7b1bf2d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/636897ff0a85c7e3e55a3bb1f8f79ae33a4f9e39d8c35e23151a523e7b1bf2d4/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c8f10c4848ba9278ffe38fe6eeebdf1bcf4f34fa889e63eb568816b6b543af8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c8f10c4848ba9278ffe38fe6eeebdf1bcf4f34fa889e63eb568816b6b543af8/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/809989d2cacca82b7944f52ffbc7a90dc664b50616965de57f062a5f95b8bff8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/809989d2cacca82b7944f52ffbc7a90dc664b50616965de57f062a5f95b8bff8/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/832fa605d5ce18b179ff8c4e2c15029b60c9003af1371f9cc5624a3cbb6b1976/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/832fa605d5ce18b179ff8c4e2c15029b60c9003af1371f9cc5624a3cbb6b1976/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8df58a0005cdf247c07c538f0ecb2e97ca6d81abff60714e004d080e9c579a9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8df58a0005cdf247c07c538f0ecb2e97ca6d81abff60714e004d080e9c579a9e/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4f67617ed451d37c9bee2fa996dcac9e63113d20cb96ca12df1d0334d934b85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4f67617ed451d37c9bee2fa996dcac9e63113d20cb96ca12df1d0334d934b85/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2538555ee0daaad4942a4aeb2b16e83f115d481c85cae536b5706bff296827f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2538555ee0daaad4942a4aeb2b16e83f115d481c85cae536b5706bff296827f/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb0314051d7513894134c186b87d00d4f1d98174bb3362a535b1d060c0ccad66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb0314051d7513894134c186b87d00d4f1d98174bb3362a535b1d060c0ccad66/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc8168a8fa89b0a7de21570865493f7d8545df2b6a770930a417edb4cffa8676/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc8168a8fa89b0a7de21570865493f7d8545df2b6a770930a417edb4cffa8676/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6cac17a243ecfd4611217de16d529f97a77d4c9e00d3fcfa0662a1a1c8cd3f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6cac17a243ecfd4611217de16d529f97a77d4c9e00d3fcfa0662a1a1c8cd3f7/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de85c47ff6f88560baa95f2819c00fade8d284c35087bf8ad98102c40cdb279f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de85c47ff6f88560baa95f2819c00fade8d284c35087bf8ad98102c40cdb279f/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6d43012785906e18c9802e1072e7f84a90b3dd1c4ff63b59cb094da8141f808/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6d43012785906e18c9802e1072e7f84a90b3dd1c4ff63b59cb094da8141f808/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffe9267f936342b7b037e91fccd9c0e4ab92c2912899d3c0edcd8500804c87eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffe9267f936342b7b037e91fccd9c0e4ab92c2912899d3c0edcd8500804c87eb/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b/volumes/kubernetes.io~projected/kube-api-access-k8gb9:{mountpoint:/var/lib/kubelet/pods/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b/volumes/kubernetes.io~projected/kube-api-access-k8gb9 major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b/volumes/kubernetes.io~secret/signing-key major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2293549b-8f94-4bf2-8755-e83e4350fd79/volumes/kubernetes.io~projected/kube-api-access-jjmzh:{mountpoint:/var/lib/kubelet/pods/2293549b-8f94-4bf2-8755-e83e4350fd79/volumes/kubernetes.io~projected/kube-api-access-jjmzh major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/241808e4-15ac-452d-9139-85411bc375cd/volumes/kubernetes.io~projected/kube-api-access-qzd7b:{mountpoint:/var/lib/kubelet/pods/241808e4-15ac-452d-9139-85411bc375cd/volumes/kubernetes.io~projected/kube-api-access-qzd7b major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/241808e4-15ac-452d-9139-85411bc375cd/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/241808e4-15ac-452d-9139-85411bc375cd/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/373b617b-0896-420c-b2fd-38635c62a21c/volumes/kubernetes.io~projected/kube-api-access-v974s:{mountpoint:/var/lib/kubelet/pods/373b617b-0896-420c-b2fd-38635c62a21c/volumes/kubernetes.io~projected/kube-api-access-v974s major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d1a766f-5256-4044-98d5-f551f811e684/volumes/kubernetes.io~projected/kube-api-access-fgrg7:{mountpoint:/var/lib/kubelet/pods/3d1a766f-5256-4044-98d5-f551f811e684/volumes/kubernetes.io~projected/kube-api-access-fgrg7 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42b67c56-566b-4319-93ff-165d6cefdf19/volumes/kubernetes.io~projected/kube-api-access-6245n:{mountpoint:/var/lib/kubelet/pods/42b67c56-566b-4319-93ff-165d6cefdf19/volumes/kubernetes.io~projected/kube-api-access-6245n major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44f7fda2-c7d3-470c-9080-1f931fac9baa/volumes/kubernetes.io~projected/kube-api-access-jh4p5:{mountpoint:/var/lib/kubelet/pods/44f7fda2-c7d3-470c-9080-1f931fac9baa/volumes/kubernetes.io~projected/kube-api-access-jh4p5 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c/volumes/kubernetes.io~projected/kube-api-access-cw5gw:{mountpoint:/var/lib/kubelet/pods/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c/volumes/kubernetes.io~projected/kube-api-access-cw5gw major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dcbf677-ca92-4306-bc45-ab9954ed7ac2/volumes/kubernetes.io~projected/kube-api-access-c6xm9:{mountpoint:/var/lib/kubelet/pods/7dcbf677-ca92-4306-bc45-ab9954ed7ac2/volumes/kubernetes.io~projected/kube-api-access-c6xm9 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a619b2e9-c604-4179-aa69-aa822f209f0c/volumes/kubernetes.io~projected/kube-api-access-9ks47:{mountpoint:/var/lib/kubelet/pods/a619b2e9-c604-4179-aa69-aa822f209f0c/volumes/kubernetes.io~projected/kube-api-access-9ks47 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a62fb314-b483-4dcd-8170-a818757da247/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a62fb314-b483-4dcd-8170-a818757da247/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a62fb314-b483-4dcd-8170-a818757da247/volumes/kubernetes.io~projected/kube-api-access-5r5xm:{mountpoint:/var/lib/kubelet/pods/a62fb314-b483-4dcd-8170-a818757da247/volumes/kubernetes.io~projected/kube-api-access-5r5xm major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a62fb314-b483-4dcd-8170-a818757da247/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a62fb314-b483-4dcd-8170-a818757da247/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6f66f51-48bd-439d-92b7-a3c9dc33f591/volumes/kubernetes.io~projected/kube-api-access-2hlgt:{mountpoint:/var/lib/kubelet/pods/a6f66f51-48bd-439d-92b7-a3c9dc33f591/volumes/kubernetes.io~projected/kube-api-access-2hlgt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad299622-53d4-4610-b03a-3c586c0fff56/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ad299622-53d4-4610-b03a-3c586c0fff56/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd91ed10-7ecf-415f-9906-3a79c23c8229/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/bd91ed10-7ecf-415f-9906-3a79c23c8229/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd91ed10-7ecf-415f-9906-3a79c23c8229/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/bd91ed10-7ecf-415f-9906-3a79c23c8229/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd91ed10-7ecf-415f-9906-3a79c23c8229/volumes/kubernetes.io~projected/kube-api-access-wdrqv:{mountpoint:/var/lib/kubelet/pods/bd91ed10-7ecf-415f-9906-3a79c23c8229/volumes/kubernetes.io~projected/kube-api-access-wdrqv major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0/volumes/kubernetes.io~projected/kube-api-access-jq9wm:{mountpoint:/var/lib/kubelet/pods/d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0/volumes/kubernetes.io~projected/kube-api-access-jq9wm major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc918fe6-f902-419e-b4f1-04690e999f47/volumes/kubernetes.io~projected/kube-api-access-q7xhf:{mountpoint:/var/lib/kubelet/pods/dc918fe6-f902-419e-b4f1-04690e999f47/volumes/kubernetes.io~projected/kube-api-access-q7xhf major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddd92762-f7c4-4f85-80e1-0fe7045e78e8/volumes/kubernetes.io~projected/kube-api-access-qnzd6:{mountpoint:/var/lib/kubelet/pods/ddd92762-f7c4-4f85-80e1-0fe7045e78e8/volumes/kubernetes.io~projected/kube-api-access-qnzd6 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/cbc6faadab818a5b25567aa2367860036b8d7659dc141d05e3e02a14465777f2/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/38f081c7d7650308f88cf613469fe52d9ad2fc54a9fa10c6045be5662bc5019f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/335eefb9aadbcd5db490755dc42a8aa60cb23347107da45a9cb494e628ce5693/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/29ccd82a1fc0924d2a06f71109c411734711d646a99e336f2a708ed2c42075df/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/4e9b4d13efa2595bc398482c1f12bfd3db3e503b7c31b121be697e193f404fc6/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/47d860c4b8355c497cda8955fe3e8c226ed58485a8e490b2a6bc3b6ed4708bba/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d62aa600f3c1af348a9e45fa0c20704af6dda3839e7edcc73966c9868870bbef/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1c0277de57cda967108d0aec9498242fa1c52d7eb321fc8838a8944879c4f490/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/85bdf1a8840951fb60c6b9942b7ae8bfb99314f2bb4741010cc65bbb7a2f1c77/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/a53c2d277e0cd7771686f7c999b383ac0680c4ca5e2dea484be4421561c7c4c7/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/5937587fe85479d887c5c26b4939f058b0632a51ac0aee8f5be4996d3530285d/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/fc3000cc2c7d9a11c9161861dc735ff59f242f273b7dfbd5c639e717907cf603/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/b70a5ed1dda0833a47f3a17a841202792198af26aba3015fed0216a744c6a5d8/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/6211e6f0d3fff1a6a03fb8ec42f87141a8273b6dd5df9fef4af29d3ffcf904d4/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/1f08aa994a38255a062cc51279e22812a001a6cb955f95ba1e4cfd6fba7a0008/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/8f612adcb3cba03bfda645ed576896e87875f2534b928544ce973ec12969beda/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/df327d6011e9b424d0dacd946fe330c6dbcb3d93ba4f1797bc5c27a4b33a8b0c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/377173f2bf3c56b94ead5d51587ddf14a4f975dca5e48ec16c8f565584bda0cb/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ec407c8dda5b59d74a5c61b12090fe13bc730b898c2282dbf2cb5fb4e257955a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f504667bebf25ce22cfd09d8613e9c683a4bd1fde2456c83fc623033e113131d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8f68be755d227e1c88121821542876a9a64182dfff4bcfb42aabba782cd2a30d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/61e22c0c5fe117aea108fee72798cb315ca8c937e2d70d1e1ee54f217bed1cd2/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/f0d873e302b1ed81600c67b3a64705289200df9860e390ca0230494c143392e5/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/7409e3b9f336931dcc0c39fc7e6549777e5d1d19715c3eb51c89e7356d77a165/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/16ec08769419681ea1306a4b88648561115ab979f67f0b896b4421fb1d2ef213/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/193764f09a6ec51afa5aeeaeb0ef4877796dbd11843f5292f3057349b35fd406/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/39ccf23e57c363af28df208e17ffb1a96e5e69b55c7e07c8568bc10fbdced404/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/bb32124c2f508392e3250b73243fe16f0f2f85a3805d629ac5c64831f3f39f40/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/6c17617ddfd7c29cd79ec312e4b755415c865c38f5fe80afdb31250cc6b4530e/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/0d4b82a2cc395b20bceea5b830b10b634ecfb2a5197df7b3c5559fe132d3c638/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/36214e0ea6a8b00cc24a426079efa73bb7d986dab54daaf764cabaa289054a64/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/16e19317298a4af8676088ede69af7b49792d1c86de7595f3db5c521ede3fb02/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/273b615205167d44bc8c36bdfda650c45d11f9f94bd76934fd5578c48ba20c3f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2c2f047deb694dee619b4e545026747c08269d73c8116fbd4c307659faa9b44f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/ca90071839b12936e1d1bb6e2d1c559b8c022b466654bd13494cd63323cec23c/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/5b3d3abd3f04bd88ea731a8e79983fd3001a9a4ec0577926de637dc4b4353a9d/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/16c3535e64a9acdb182816fca6fdacde3fe423bdac0e5095db03218a5b36664e/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d03fd44bf71c87926419efd6787461e7038ce2e65cd9ee38d423e0caf7e296b0/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/51271331599a1638c8c22263c887536b4ed55620b8442c2d65c1efc525c97ab9/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/1db97969a1f5f55c322ed7121c40ae02ec8ac9c5014c26708b2ad45550d12f36/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8b8a0eec616fc9177275f6078add54f965b26c3a1210c09666ac2acdcd130258/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.150592 4433 manager.go:217] Machine: {Timestamp:2026-10-30 19:15:53.149928512 +0000 UTC m=+0.084716033 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199796 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2906c43bfb44990648ec3708beb21b SystemUUID:ec2906c4-3bfb-4499-0648-ec3708beb21b BootID:6cc2a45c-6bf0-4e14-a980-eb98357404ad Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/dc918fe6-f902-419e-b4f1-04690e999f47/volumes/kubernetes.io~projected/kube-api-access-q7xhf DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:235 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c8f10c4848ba9278ffe38fe6eeebdf1bcf4f34fa889e63eb568816b6b543af8/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/bd91ed10-7ecf-415f-9906-3a79c23c8229/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a62fb314-b483-4dcd-8170-a818757da247/volumes/kubernetes.io~projected/kube-api-access-5r5xm DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de85c47ff6f88560baa95f2819c00fade8d284c35087bf8ad98102c40cdb279f/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb0314051d7513894134c186b87d00d4f1d98174bb3362a535b1d060c0ccad66/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a62fb314-b483-4dcd-8170-a818757da247/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2293549b-8f94-4bf2-8755-e83e4350fd79/volumes/kubernetes.io~projected/kube-api-access-jjmzh DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/241808e4-15ac-452d-9139-85411bc375cd/volumes/kubernetes.io~projected/kube-api-access-qzd7b DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6f66f51-48bd-439d-92b7-a3c9dc33f591/volumes/kubernetes.io~projected/kube-api-access-2hlgt DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6cac17a243ecfd4611217de16d529f97a77d4c9e00d3fcfa0662a1a1c8cd3f7/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d1a766f-5256-4044-98d5-f551f811e684/volumes/kubernetes.io~projected/kube-api-access-fgrg7 DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0/volumes/kubernetes.io~projected/kube-api-access-jq9wm DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c/volumes/kubernetes.io~projected/kube-api-access-cw5gw DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b/volumes/kubernetes.io~projected/kube-api-access-k8gb9 DeviceMajor:0 DeviceMinor:240 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6d43012785906e18c9802e1072e7f84a90b3dd1c4ff63b59cb094da8141f808/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/809989d2cacca82b7944f52ffbc7a90dc664b50616965de57f062a5f95b8bff8/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ddd92762-f7c4-4f85-80e1-0fe7045e78e8/volumes/kubernetes.io~projected/kube-api-access-qnzd6 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00f7c9795076d2276b5d53c2f77ece72abdba65e4e5b47dcd39cd0350e710ff8/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/832fa605d5ce18b179ff8c4e2c15029b60c9003af1371f9cc5624a3cbb6b1976/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad299622-53d4-4610-b03a-3c586c0fff56/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bd91ed10-7ecf-415f-9906-3a79c23c8229/volumes/kubernetes.io~projected/kube-api-access-wdrqv DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/373b617b-0896-420c-b2fd-38635c62a21c/volumes/kubernetes.io~projected/kube-api-access-v974s DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a619b2e9-c604-4179-aa69-aa822f209f0c/volumes/kubernetes.io~projected/kube-api-access-9ks47 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/241808e4-15ac-452d-9139-85411bc375cd/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23d24245e8b069e7f3cf92d29fbcd4bf6ac16b18cd49bd593e33b0916615cff3/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/bd91ed10-7ecf-415f-9906-3a79c23c8229/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8df58a0005cdf247c07c538f0ecb2e97ca6d81abff60714e004d080e9c579a9e/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc8168a8fa89b0a7de21570865493f7d8545df2b6a770930a417edb4cffa8676/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a62fb314-b483-4dcd-8170-a818757da247/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/7dcbf677-ca92-4306-bc45-ab9954ed7ac2/volumes/kubernetes.io~projected/kube-api-access-c6xm9 DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1eee746462d68b33b3b02fe2ac8ca0126457fa5e52540af045fe141565c717a5/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4f67617ed451d37c9bee2fa996dcac9e63113d20cb96ca12df1d0334d934b85/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44f7fda2-c7d3-470c-9080-1f931fac9baa/volumes/kubernetes.io~projected/kube-api-access-jh4p5 DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffe9267f936342b7b037e91fccd9c0e4ab92c2912899d3c0edcd8500804c87eb/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/636897ff0a85c7e3e55a3bb1f8f79ae33a4f9e39d8c35e23151a523e7b1bf2d4/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/42b67c56-566b-4319-93ff-165d6cefdf19/volumes/kubernetes.io~projected/kube-api-access-6245n DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2538555ee0daaad4942a4aeb2b16e83f115d481c85cae536b5706bff296827f/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:23d24245e8b069e MacAddress:7e:57:fb:5b:a9:69 Speed:10000 Mtu:8901} {Name:636897ff0a85c7e MacAddress:2e:16:10:dd:0a:40 Speed:10000 Mtu:8901} {Name:6c8f10c4848ba92 MacAddress:12:42:fb:99:83:4a Speed:10000 Mtu:8901} {Name:8df58a0005cdf24 MacAddress:36:54:ec:f6:f3:26 Speed:10000 Mtu:8901} {Name:b2538555ee0daaa MacAddress:ce:96:5f:fc:f8:5c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:58:f0:f6:81:39 Speed:0 Mtu:9001} {Name:br-int MacAddress:76:5c:97:04:ba:94 Speed:0 Mtu:8901} {Name:e6d43012785906e MacAddress:1a:37:02:74:a5:12 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:58:f0:f6:81:39 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:8a:89:2e:f6:6d:34 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:f2:71:ee:dd:5c:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.150886 4433 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.150934 4433 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.151234 4433 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.151259 4433 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-5.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.151373 4433 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.151381 4433 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.151390 4433 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.151408 4433 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.151485 4433 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.151655 4433 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.151757 4433 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.151789 4433 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.151798 4433 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.151808 4433 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.151817 4433 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.151824 4433 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.152440 4433 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.152449 4433 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:15:53.155885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.152945 4433 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:15:53.162424 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.162407 4433 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:15:53.162424 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.162424 4433 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:15:53.162560 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.162524 4433 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 19:15:53.162778 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.162751 4433 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:15:53.162842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.162787 4433 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:15:53.162900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.162888 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:15:53.162942 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.162909 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:15:53.162942 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.162920 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:15:53.162942 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.162928 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:15:53.162942 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.162939 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:15:53.163146 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.162947 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:15:53.163146 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.162959 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:15:53.163146 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.162967 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:15:53.163146 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.162974 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:15:53.163146 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.162996 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:15:53.163146 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.163009 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:15:53.163146 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.163023 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:15:53.163146 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.163050 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:15:53.163146 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.163059 4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:15:53.168078 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.167236 4433 server.go:1263] "Started kubelet" Oct 30 19:15:53.168078 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.167669 4433 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:15:53.168078 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.167784 4433 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:15:53.167744 ip-10-0-141-5 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:15:53.168632 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.168106 4433 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:15:53.168632 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.168151 4433 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:15:53.169208 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.169189 4433 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:15:53.175391 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.175298 4433 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:15:53.175936 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:53.175898 4433 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-5.us-west-1.compute.internal\" not found" Oct 30 19:15:53.176398 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.176323 4433 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:15:53.176398 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.176342 4433 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:15:53.176529 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.176409 4433 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:15:53.176529 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.176421 4433 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:15:53.176529 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.176497 4433 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:10:15 +0000 UTC" deadline="2028-06-12 03:22:20.90515917 +0000 UTC" Oct 30 19:15:53.176529 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.176518 4433 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14168h6m27.728643954s" Oct 30 19:15:53.177072 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.176916 4433 factory.go:55] Registering systemd factory Oct 30 19:15:53.177072 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.176937 4433 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:15:53.177233 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.177193 4433 factory.go:153] Registering CRI-O factory Oct 30 19:15:53.177233 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.177210 4433 factory.go:223] Registration of the crio container factory successfully Oct 30 19:15:53.177334 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.177260 4433 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:15:53.177334 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.177280 4433 factory.go:103] Registering Raw factory Oct 30 19:15:53.177334 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.177293 4433 manager.go:1199] Started watching for new ooms in manager Oct 30 19:15:53.177676 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.177660 4433 manager.go:319] Starting recovery of all containers Oct 30 19:15:53.178040 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.177973 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="241808e4-15ac-452d-9139-85411bc375cd" volumeName="kubernetes.io/projected/241808e4-15ac-452d-9139-85411bc375cd-kube-api-access-qzd7b" seLinuxMountContext="" Oct 30 19:15:53.178136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178041 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="373b617b-0896-420c-b2fd-38635c62a21c" volumeName="kubernetes.io/configmap/373b617b-0896-420c-b2fd-38635c62a21c-serviceca" seLinuxMountContext="" Oct 30 19:15:53.178136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178058 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69a39e1d-7ab3-40a0-9f5d-ae469a5d326c" volumeName="kubernetes.io/empty-dir/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-data-volume" seLinuxMountContext="" Oct 30 19:15:53.178136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178072 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dcbf677-ca92-4306-bc45-ab9954ed7ac2" volumeName="kubernetes.io/configmap/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-config-volume" seLinuxMountContext="" Oct 30 19:15:53.178136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178083 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d921aae-0b5b-4f3f-9ad5-33fcacb7213b" volumeName="kubernetes.io/configmap/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-signing-cabundle" seLinuxMountContext="" Oct 30 19:15:53.178136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178095 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="241808e4-15ac-452d-9139-85411bc375cd" volumeName="kubernetes.io/secret/241808e4-15ac-452d-9139-85411bc375cd-cert" seLinuxMountContext="" Oct 30 19:15:53.178136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178107 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44f7fda2-c7d3-470c-9080-1f931fac9baa" volumeName="kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:15:53.178136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178119 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69a39e1d-7ab3-40a0-9f5d-ae469a5d326c" volumeName="kubernetes.io/projected/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-api-access-cw5gw" seLinuxMountContext="" Oct 30 19:15:53.178136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178134 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6f66f51-48bd-439d-92b7-a3c9dc33f591" volumeName="kubernetes.io/configmap/a6f66f51-48bd-439d-92b7-a3c9dc33f591-cni-binary-copy" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178147 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd91ed10-7ecf-415f-9906-3a79c23c8229" volumeName="kubernetes.io/projected/bd91ed10-7ecf-415f-9906-3a79c23c8229-kube-api-access-wdrqv" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178160 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddd92762-f7c4-4f85-80e1-0fe7045e78e8" volumeName="kubernetes.io/empty-dir/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-tmp-dir" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178173 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dcbf677-ca92-4306-bc45-ab9954ed7ac2" volumeName="kubernetes.io/projected/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-kube-api-access-c6xm9" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178184 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6f66f51-48bd-439d-92b7-a3c9dc33f591" volumeName="kubernetes.io/projected/a6f66f51-48bd-439d-92b7-a3c9dc33f591-kube-api-access-2hlgt" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178196 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0" volumeName="kubernetes.io/projected/d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0-kube-api-access-jq9wm" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178212 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d921aae-0b5b-4f3f-9ad5-33fcacb7213b" volumeName="kubernetes.io/projected/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-kube-api-access-k8gb9" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178224 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2293549b-8f94-4bf2-8755-e83e4350fd79" volumeName="kubernetes.io/projected/2293549b-8f94-4bf2-8755-e83e4350fd79-kube-api-access-jjmzh" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178237 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="373b617b-0896-420c-b2fd-38635c62a21c" volumeName="kubernetes.io/projected/373b617b-0896-420c-b2fd-38635c62a21c-kube-api-access-v974s" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178250 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d1a766f-5256-4044-98d5-f551f811e684" volumeName="kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178261 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44f7fda2-c7d3-470c-9080-1f931fac9baa" volumeName="kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-service-ca-bundle" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178273 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44f7fda2-c7d3-470c-9080-1f931fac9baa" volumeName="kubernetes.io/empty-dir/44f7fda2-c7d3-470c-9080-1f931fac9baa-archives-path" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178285 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a619b2e9-c604-4179-aa69-aa822f209f0c" volumeName="kubernetes.io/projected/a619b2e9-c604-4179-aa69-aa822f209f0c-kube-api-access-9ks47" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178297 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a62fb314-b483-4dcd-8170-a818757da247" volumeName="kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-ovnkube-script-lib" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178308 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a62fb314-b483-4dcd-8170-a818757da247" volumeName="kubernetes.io/projected/a62fb314-b483-4dcd-8170-a818757da247-kube-api-access-5r5xm" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178345 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a62fb314-b483-4dcd-8170-a818757da247" volumeName="kubernetes.io/secret/a62fb314-b483-4dcd-8170-a818757da247-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178359 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd91ed10-7ecf-415f-9906-3a79c23c8229" volumeName="kubernetes.io/empty-dir/bd91ed10-7ecf-415f-9906-3a79c23c8229-tmp" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178371 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc918fe6-f902-419e-b4f1-04690e999f47" volumeName="kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178384 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed1dc05f-2a13-4dd2-9d20-f5e96f17099f" volumeName="kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178396 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d921aae-0b5b-4f3f-9ad5-33fcacb7213b" volumeName="kubernetes.io/secret/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-signing-key" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178439 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2293549b-8f94-4bf2-8755-e83e4350fd79" volumeName="kubernetes.io/configmap/2293549b-8f94-4bf2-8755-e83e4350fd79-iptables-alerter-script" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178454 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42b67c56-566b-4319-93ff-165d6cefdf19" volumeName="kubernetes.io/projected/42b67c56-566b-4319-93ff-165d6cefdf19-kube-api-access-6245n" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178469 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44f7fda2-c7d3-470c-9080-1f931fac9baa" volumeName="kubernetes.io/projected/44f7fda2-c7d3-470c-9080-1f931fac9baa-kube-api-access-jh4p5" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178482 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad299622-53d4-4610-b03a-3c586c0fff56" volumeName="kubernetes.io/secret/ad299622-53d4-4610-b03a-3c586c0fff56-agent-certs" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178496 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd91ed10-7ecf-415f-9906-3a79c23c8229" volumeName="kubernetes.io/empty-dir/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-tuned" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178508 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc918fe6-f902-419e-b4f1-04690e999f47" volumeName="kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178520 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc918fe6-f902-419e-b4f1-04690e999f47" volumeName="kubernetes.io/projected/dc918fe6-f902-419e-b4f1-04690e999f47-kube-api-access-q7xhf" seLinuxMountContext="" Oct 30 19:15:53.178526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178532 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dcbf677-ca92-4306-bc45-ab9954ed7ac2" volumeName="kubernetes.io/empty-dir/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-tmp-dir" seLinuxMountContext="" Oct 30 19:15:53.179686 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178545 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a62fb314-b483-4dcd-8170-a818757da247" volumeName="kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-ovnkube-config" seLinuxMountContext="" Oct 30 19:15:53.179686 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178558 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a62fb314-b483-4dcd-8170-a818757da247" volumeName="kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-env-overrides" seLinuxMountContext="" Oct 30 19:15:53.179686 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178571 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6f66f51-48bd-439d-92b7-a3c9dc33f591" volumeName="kubernetes.io/configmap/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-daemon-config" seLinuxMountContext="" Oct 30 19:15:53.179686 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178582 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad299622-53d4-4610-b03a-3c586c0fff56" volumeName="kubernetes.io/configmap/ad299622-53d4-4610-b03a-3c586c0fff56-konnectivity-ca" seLinuxMountContext="" Oct 30 19:15:53.179686 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178594 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc918fe6-f902-419e-b4f1-04690e999f47" volumeName="kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-cni-binary-copy" seLinuxMountContext="" Oct 30 19:15:53.179686 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178605 4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddd92762-f7c4-4f85-80e1-0fe7045e78e8" volumeName="kubernetes.io/projected/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-kube-api-access-qnzd6" seLinuxMountContext="" Oct 30 19:15:53.179686 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178616 4433 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:15:53.179686 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.178623 4433 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:15:53.204012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.203952 4433 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:15:53.205581 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.205555 4433 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:15:53.205581 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.205577 4433 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:15:53.205738 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.205600 4433 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:15:53.205738 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:53.205641 4433 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 19:15:53.243115 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.243073 4433 generic.go:356] "Generic (PLEG): container finished" podID="dc918fe6-f902-419e-b4f1-04690e999f47" containerID="ab085e856154b4612b87cb2ef40f5a932121cc317329c98b545a663cfc1406b3" exitCode=0 Oct 30 19:15:53.243115 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.243113 4433 generic.go:356] "Generic (PLEG): container finished" podID="dc918fe6-f902-419e-b4f1-04690e999f47" containerID="535dde6497a5208aa3f4c9db6f1e1882fdd41b7886e78af6e7d52437a5a21098" exitCode=0 Oct 30 19:15:53.243319 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.243124 4433 generic.go:356] "Generic (PLEG): container finished" podID="dc918fe6-f902-419e-b4f1-04690e999f47" containerID="1b083fc73a549a823599cfaa2c37af0c09e5e12dbd3341f36d723af3b2352689" exitCode=0 Oct 30 19:15:53.243319 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.243135 4433 generic.go:356] "Generic (PLEG): container finished" podID="dc918fe6-f902-419e-b4f1-04690e999f47" containerID="0ff215eceaa445823ea8225f39894e2688ac507b5d9aa004cc3864259a662f64" exitCode=0 Oct 30 19:15:53.243319 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.243140 4433 generic.go:356] "Generic (PLEG): container finished" podID="dc918fe6-f902-419e-b4f1-04690e999f47" containerID="0f50efc1235efc95c66811ead4939c8278d38048d262900e9e294a3cbc3ceda0" exitCode=0 Oct 30 19:15:53.243319 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.243147 4433 generic.go:356] "Generic (PLEG): container finished" podID="dc918fe6-f902-419e-b4f1-04690e999f47" containerID="c276a97839d8a209e11376eaec34037f86127aaec5b3f2a49e5e2a6e94149650" exitCode=0 Oct 30 19:15:53.248927 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.248906 4433 manager.go:324] Recovery completed Oct 30 19:15:53.250156 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.250129 4433 generic.go:356] "Generic (PLEG): container finished" podID="56c484cac7643547da8de4853003de0e" containerID="3ee51a9598682182e4d6661d583ea5d95b56b6ea1fd1412b0e80957b5ccfd965" exitCode=0 Oct 30 19:15:53.263608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.263556 4433 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:53.264601 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.264582 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:53.264696 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.264672 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:53.264747 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.264697 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:53.266183 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266168 4433 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:15:53.266183 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266181 4433 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:15:53.266320 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266189 4433 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:53.266320 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266201 4433 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:15:53.266474 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266348 4433 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 19:15:53.266474 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266363 4433 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 19:15:53.266474 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266396 4433 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 19:15:53.266474 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266404 4433 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 19:15:53.266474 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266411 4433 policy_none.go:50] "Start" Oct 30 19:15:53.266474 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266422 4433 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:15:53.266474 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266436 4433 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:15:53.266775 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266617 4433 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 19:15:53.266775 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266629 4433 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 19:15:53.266775 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266636 4433 policy_none.go:44] "Start" Oct 30 19:15:53.266951 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266933 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:53.267067 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266964 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:53.267067 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.266995 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:53.270435 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.270411 4433 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:15:53.273308 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.273293 4433 manager.go:341] "Starting Device Plugin manager" Oct 30 19:15:53.273386 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:53.273316 4433 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:15:53.273386 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.273325 4433 server.go:87] "Starting device plugin registration server" Oct 30 19:15:53.273568 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.273555 4433 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:15:53.273613 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.273569 4433 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:15:53.273679 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.273661 4433 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:15:53.273788 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.273756 4433 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:15:53.273836 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.273797 4433 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:15:53.273836 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.273803 4433 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:15:53.273934 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.273848 4433 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:15:53.273786995Z","UUID":"d375f10c-63b3-4570-8f06-a943667cbedc","Handler":null,"Name":"","Endpoint":""} Oct 30 19:15:53.275154 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:53.275131 4433 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:15:53.275243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.275153 4433 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:15:53.275243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.275174 4433 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:15:53.275243 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:53.275175 4433 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-5.us-west-1.compute.internal\" not found" Oct 30 19:15:53.305998 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.305932 4433 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal"] Oct 30 19:15:53.306089 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.306069 4433 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:53.306840 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.306822 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:53.306939 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.306846 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:53.306939 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.306857 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:53.307108 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.307098 4433 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:53.307266 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.307251 4433 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:53.307811 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.307795 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:53.307811 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.307804 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:53.307930 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.307826 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:53.307930 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.307838 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:53.307930 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.307864 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:53.307930 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.307840 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:53.308066 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:53.308016 4433 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-5.us-west-1.compute.internal\" not found" node="ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.308196 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.308168 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" event={"ID":"56c484cac7643547da8de4853003de0e","Type":"ContainerStarted","Data":"772b2159470cd609c9ded161021dea72191c1d9016efa1d580987bda83459b5d"} Oct 30 19:15:53.308230 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.308204 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" event={"ID":"56c484cac7643547da8de4853003de0e","Type":"ContainerDied","Data":"3ee51a9598682182e4d6661d583ea5d95b56b6ea1fd1412b0e80957b5ccfd965"} Oct 30 19:15:53.308230 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.308216 4433 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:53.308295 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.308217 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" event={"ID":"56c484cac7643547da8de4853003de0e","Type":"ContainerStarted","Data":"ffe9267f936342b7b037e91fccd9c0e4ab92c2912899d3c0edcd8500804c87eb"} Oct 30 19:15:53.308345 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.308300 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" event={"ID":"d17eaea73d7bf32a8a9bb9e63451f939","Type":"ContainerStarted","Data":"c0b36c4262aca8f337d1546c449c83549d970509e0285b699d352906e971194d"} Oct 30 19:15:53.308345 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.308313 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" event={"ID":"d17eaea73d7bf32a8a9bb9e63451f939","Type":"ContainerStarted","Data":"809989d2cacca82b7944f52ffbc7a90dc664b50616965de57f062a5f95b8bff8"} Oct 30 19:15:53.308876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.308861 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:53.308937 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.308891 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:53.308937 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.308901 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:53.309035 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:53.309024 4433 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-5.us-west-1.compute.internal\" not found" node="ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.361447 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.361425 4433 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:15:53.361611 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.361597 4433 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 19:15:53.362217 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.362201 4433 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:15:53.362576 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.362561 4433 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:15:53.373869 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.373855 4433 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:53.375180 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.375164 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:53.375241 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.375189 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:53.375241 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.375199 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:53.375241 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.375236 4433 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.377299 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.377284 4433 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.412354 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.412319 4433 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:15:53.458942 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.458924 4433 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:15:53.459027 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:53.458960 4433 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.459027 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.458972 4433 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.462091 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.462073 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d17eaea73d7bf32a8a9bb9e63451f939-config\") pod \"kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"d17eaea73d7bf32a8a9bb9e63451f939\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.462149 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.462100 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56c484cac7643547da8de4853003de0e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"56c484cac7643547da8de4853003de0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.462149 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.462117 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56c484cac7643547da8de4853003de0e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"56c484cac7643547da8de4853003de0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.485045 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.485025 4433 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.485108 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.485086 4433 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.485573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.485559 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeNotReady" Oct 30 19:15:53.485656 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.485589 4433 setters.go:546] "Node became not ready" node="ip-10-0-141-5.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T19:15:53Z","lastTransitionTime":"2026-10-30T19:15:53Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 30 19:15:53.518700 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.518649 4433 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:15:53.518700 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:53.518687 4433 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.535575 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.535551 4433 nodeinfomanager.go:471] existing CSINode "ip-10-0-141-5.us-west-1.compute.internal" is owned by different node (oldNodeID="a8d610b3-915e-4469-9a9c-bdec13501922", newNodeID=""), cleaning up... Oct 30 19:15:53.562775 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.562752 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56c484cac7643547da8de4853003de0e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"56c484cac7643547da8de4853003de0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.562871 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.562778 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56c484cac7643547da8de4853003de0e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"56c484cac7643547da8de4853003de0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.562871 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.562793 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d17eaea73d7bf32a8a9bb9e63451f939-config\") pod \"kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"d17eaea73d7bf32a8a9bb9e63451f939\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.562871 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.562840 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d17eaea73d7bf32a8a9bb9e63451f939-config\") pod \"kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"d17eaea73d7bf32a8a9bb9e63451f939\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.562871 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.562851 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56c484cac7643547da8de4853003de0e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"56c484cac7643547da8de4853003de0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.562871 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.562858 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56c484cac7643547da8de4853003de0e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal\" (UID: \"56c484cac7643547da8de4853003de0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:53.852122 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.852101 4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-5.us-west-1.compute.internal" event="NodeReady" Oct 30 19:15:53.852262 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:53.852163 4433 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 19:15:54.152310 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.152252 4433 apiserver.go:52] "Watching apiserver" Oct 30 19:15:54.255974 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.255945 4433 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:54.256219 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.256112 4433 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:54.257298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.257280 4433 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:15:54.257775 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.257751 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-m92f6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w","openshift-dns/dns-default-4j2jg","openshift-multus/network-metrics-daemon-7gj92","openshift-network-diagnostics/network-check-target-ql75b","kube-system/konnectivity-agent-lv4mc","kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal","openshift-dns/node-resolver-25zlr","openshift-network-operator/iptables-alerter-lzl7q","openshift-ovn-kubernetes/ovnkube-node-kg9n8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal","openshift-multus/multus-kldq6","kube-system/global-pull-secret-syncer-6vks7","openshift-ingress-canary/ingress-canary-rp9mn","openshift-insights/periodic-gathering-hjklm-r9jn7","openshift-cluster-node-tuning-operator/tuned-8ch84","openshift-image-registry/node-ca-s4k9z","openshift-kube-storage-version-migrator/migrator-66df5d496b-fs446","openshift-multus/multus-additional-cni-plugins-v2hmn","openshift-insights/insights-runtime-extractor-hjh7f"] Oct 30 19:15:54.262102 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.262079 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:54.262479 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.262459 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:54.263648 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.263632 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:54.278391 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.278374 4433 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:15:54.313966 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.313939 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:15:54.317596 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.317578 4433 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:15:54.317670 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:54.317626 4433 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:54.317670 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.317584 4433 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:15:54.317760 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:54.317698 4433 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-5.us-west-1.compute.internal" Oct 30 19:15:54.318247 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.318229 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4h2hd\"" Oct 30 19:15:54.318419 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.318404 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:15:54.318419 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.318413 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-csh58\"" Oct 30 19:15:54.318601 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.318428 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6brtl\"" Oct 30 19:15:54.318601 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.318451 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:15:54.318601 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.318475 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hk6fn\"" Oct 30 19:15:54.318601 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.318476 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4qn9x\"" Oct 30 19:15:54.318601 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.318475 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rhxwm\"" Oct 30 19:15:54.318601 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.318501 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:15:54.318601 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.318517 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4k9jz\"" Oct 30 19:15:54.318994 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.318668 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:15:54.318994 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.318724 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q7svh\"" Oct 30 19:15:54.318994 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.318758 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jlfgq\"" Oct 30 19:15:54.318994 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.318861 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:15:54.318994 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.318870 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gt8zl\"" Oct 30 19:15:54.318994 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.318935 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-px7h6\"" Oct 30 19:15:54.319205 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.319018 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n57d7\"" Oct 30 19:15:54.319205 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.319039 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5vtr8\"" Oct 30 19:15:54.327003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.326975 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:15:54.327259 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.327244 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 19:15:54.327321 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.327244 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:15:54.327543 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.327529 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 19:15:54.327790 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.327773 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:15:54.327878 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.327789 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:15:54.327878 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.327795 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:15:54.327878 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.327829 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:15:54.327878 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.327834 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:15:54.327878 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.327858 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:15:54.327878 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.327869 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:15:54.328128 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.327870 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:15:54.328128 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.328083 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:15:54.328431 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.328408 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:15:54.328431 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.328413 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:15:54.328542 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.328482 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:15:54.362871 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.362844 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:15:54.362871 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.362855 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:15:54.362968 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.362878 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:15:54.363108 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.363095 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:15:54.363148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.363139 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:15:54.366251 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.366238 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:15:54.366318 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.366261 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:15:54.366550 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.366538 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:15:54.366823 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.366810 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:15:54.366921 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.366837 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:15:54.366921 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.366842 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:15:54.366921 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.366838 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:15:54.368195 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368171 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 19:15:54.368195 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368171 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:15:54.368382 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368367 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw5gw\" (UniqueName: \"kubernetes.io/projected/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-api-access-cw5gw\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:54.368441 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368396 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/373b617b-0896-420c-b2fd-38635c62a21c-serviceca\") pod \"node-ca-s4k9z\" (UID: \"373b617b-0896-420c-b2fd-38635c62a21c\") " pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:54.368441 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368420 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6f66f51-48bd-439d-92b7-a3c9dc33f591-cni-binary-copy\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.368547 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368446 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-cni-multus\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.368547 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368504 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq9wm\" (UniqueName: \"kubernetes.io/projected/d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0-kube-api-access-jq9wm\") pod \"migrator-66df5d496b-fs446\" (UID: \"d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fs446" Oct 30 19:15:54.368547 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368532 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.368702 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368564 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2293549b-8f94-4bf2-8755-e83e4350fd79-host-slash\") pod \"iptables-alerter-lzl7q\" (UID: \"2293549b-8f94-4bf2-8755-e83e4350fd79\") " pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:54.368702 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368635 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-hostroot\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.368702 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368667 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-slash\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.368854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368701 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a62fb314-b483-4dcd-8170-a818757da247-ovn-node-metrics-cert\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.368854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368725 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad299622-53d4-4610-b03a-3c586c0fff56-konnectivity-ca\") pod \"konnectivity-agent-lv4mc\" (UID: \"ad299622-53d4-4610-b03a-3c586c0fff56\") " pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:15:54.368854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368737 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/373b617b-0896-420c-b2fd-38635c62a21c-serviceca\") pod \"node-ca-s4k9z\" (UID: \"373b617b-0896-420c-b2fd-38635c62a21c\") " pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:54.368854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368749 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/241808e4-15ac-452d-9139-85411bc375cd-cert\") pod \"ingress-canary-rp9mn\" (UID: \"241808e4-15ac-452d-9139-85411bc375cd\") " pod="openshift-ingress-canary/ingress-canary-rp9mn" Oct 30 19:15:54.368854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368774 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:54.368854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368792 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6f66f51-48bd-439d-92b7-a3c9dc33f591-cni-binary-copy\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.368854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368811 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:54.368854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368843 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.369224 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368903 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysconfig\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.369224 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.368928 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a62fb314-b483-4dcd-8170-a818757da247-ovn-node-metrics-cert\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.369224 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369023 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysctl-d\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.369224 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369067 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-lib-modules\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.369224 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369095 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-cnibin\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.369224 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369117 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-run-netns\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.369224 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369122 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad299622-53d4-4610-b03a-3c586c0fff56-konnectivity-ca\") pod \"konnectivity-agent-lv4mc\" (UID: \"ad299622-53d4-4610-b03a-3c586c0fff56\") " pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:15:54.369224 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369134 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8gb9\" (UniqueName: \"kubernetes.io/projected/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-kube-api-access-k8gb9\") pod \"service-ca-5fffc97bdf-m92f6\" (UID: \"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:54.369224 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369202 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-socket-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.369533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369242 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6xm9\" (UniqueName: \"kubernetes.io/projected/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-kube-api-access-c6xm9\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:54.369533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369279 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-system-cni-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.369533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369309 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-multus-certs\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.369533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369333 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-log-socket\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.369533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369402 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-dbus\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:54.369533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369430 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-systemd\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.369533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369457 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-registration-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.369533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369475 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-netns\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.369842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369532 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-var-lib-kubelet\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.369842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369561 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-system-cni-dir\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.369842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369582 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-tmp-dir\") pod \"node-resolver-25zlr\" (UID: \"ddd92762-f7c4-4f85-80e1-0fe7045e78e8\") " pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:54.369842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369605 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-conf-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.369842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369652 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-tmp-dir\") pod \"node-resolver-25zlr\" (UID: \"ddd92762-f7c4-4f85-80e1-0fe7045e78e8\") " pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:54.369842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369699 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hlgt\" (UniqueName: \"kubernetes.io/projected/a6f66f51-48bd-439d-92b7-a3c9dc33f591-kube-api-access-2hlgt\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.369842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369727 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.369842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369743 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-kubelet-config\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:54.369842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369757 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-device-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.369842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369771 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-run\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.369842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369803 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/373b617b-0896-420c-b2fd-38635c62a21c-host\") pod \"node-ca-s4k9z\" (UID: \"373b617b-0896-420c-b2fd-38635c62a21c\") " pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:54.369842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369818 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-cnibin\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.369842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369832 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-ovn\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.370324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369859 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-ovnkube-script-lib\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.370324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369893 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.370324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.369932 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-config-volume\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:54.370324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370024 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-systemd-units\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.370324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370051 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrg7\" (UniqueName: \"kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7\") pod \"network-check-target-ql75b\" (UID: \"3d1a766f-5256-4044-98d5-f551f811e684\") " pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:54.370324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370075 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-ovnkube-script-lib\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.370324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370075 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/44f7fda2-c7d3-470c-9080-1f931fac9baa-archives-path\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:54.370324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370131 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.370324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370141 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/44f7fda2-c7d3-470c-9080-1f931fac9baa-archives-path\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:54.370324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370176 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-config-volume\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:54.370324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370207 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:54.370324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370245 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6245n\" (UniqueName: \"kubernetes.io/projected/42b67c56-566b-4319-93ff-165d6cefdf19-kube-api-access-6245n\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.370324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370271 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.370324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370296 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-systemd\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.370324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370320 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-etc-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.370779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370359 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.370779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370388 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-ovnkube-config\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.370779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370443 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-signing-key\") pod \"service-ca-5fffc97bdf-m92f6\" (UID: \"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:54.370779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370504 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysctl-conf\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.370779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370521 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-sys-fs\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.370779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370537 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.370779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370552 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:54.370779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370559 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-ovnkube-config\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.370779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370568 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-cni-bin\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.370779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370596 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-var-lib-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.370779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370665 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzd7b\" (UniqueName: \"kubernetes.io/projected/241808e4-15ac-452d-9139-85411bc375cd-kube-api-access-qzd7b\") pod \"ingress-canary-rp9mn\" (UID: \"241808e4-15ac-452d-9139-85411bc375cd\") " pod="openshift-ingress-canary/ingress-canary-rp9mn" Oct 30 19:15:54.370779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370702 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-tmp-dir\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:54.370779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370764 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnzd6\" (UniqueName: \"kubernetes.io/projected/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-kube-api-access-qnzd6\") pod \"node-resolver-25zlr\" (UID: \"ddd92762-f7c4-4f85-80e1-0fe7045e78e8\") " pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:54.370779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370765 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc918fe6-f902-419e-b4f1-04690e999f47-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.371434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370831 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-tmp-dir\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:54.371434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370904 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-cni-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.371434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.370954 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-hosts-file\") pod \"node-resolver-25zlr\" (UID: \"ddd92762-f7c4-4f85-80e1-0fe7045e78e8\") " pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:54.371434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371001 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-signing-cabundle\") pod \"service-ca-5fffc97bdf-m92f6\" (UID: \"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:54.371434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371027 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd91ed10-7ecf-415f-9906-3a79c23c8229-tmp\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.371434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371119 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd91ed10-7ecf-415f-9906-3a79c23c8229-tmp\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.371434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371123 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7xhf\" (UniqueName: \"kubernetes.io/projected/dc918fe6-f902-419e-b4f1-04690e999f47-kube-api-access-q7xhf\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.371434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371166 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-socket-dir-parent\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.371434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371193 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-tuned\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.371434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371219 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjmzh\" (UniqueName: \"kubernetes.io/projected/2293549b-8f94-4bf2-8755-e83e4350fd79-kube-api-access-jjmzh\") pod \"iptables-alerter-lzl7q\" (UID: \"2293549b-8f94-4bf2-8755-e83e4350fd79\") " pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:54.371434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371260 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-k8s-cni-cncf-io\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.371434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371292 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-node-log\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.371434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371294 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-tuned\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.371434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371396 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.371434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371433 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-env-overrides\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371477 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad299622-53d4-4610-b03a-3c586c0fff56-agent-certs\") pod \"konnectivity-agent-lv4mc\" (UID: \"ad299622-53d4-4610-b03a-3c586c0fff56\") " pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371564 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-service-ca-bundle\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371593 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a62fb314-b483-4dcd-8170-a818757da247-env-overrides\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371621 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-sys\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371637 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371654 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-daemon-config\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371671 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-kubelet\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371692 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5xm\" (UniqueName: \"kubernetes.io/projected/a62fb314-b483-4dcd-8170-a818757da247-kube-api-access-5r5xm\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371694 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad299622-53d4-4610-b03a-3c586c0fff56-agent-certs\") pod \"konnectivity-agent-lv4mc\" (UID: \"ad299622-53d4-4610-b03a-3c586c0fff56\") " pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371736 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-modprobe-d\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371757 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdrqv\" (UniqueName: \"kubernetes.io/projected/bd91ed10-7ecf-415f-9906-3a79c23c8229-kube-api-access-wdrqv\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371771 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2293549b-8f94-4bf2-8755-e83e4350fd79-iptables-alerter-script\") pod \"iptables-alerter-lzl7q\" (UID: \"2293549b-8f94-4bf2-8755-e83e4350fd79\") " pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371794 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v974s\" (UniqueName: \"kubernetes.io/projected/373b617b-0896-420c-b2fd-38635c62a21c-kube-api-access-v974s\") pod \"node-ca-s4k9z\" (UID: \"373b617b-0896-420c-b2fd-38635c62a21c\") " pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371811 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-etc-kubernetes\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371828 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh4p5\" (UniqueName: \"kubernetes.io/projected/44f7fda2-c7d3-470c-9080-1f931fac9baa-kube-api-access-jh4p5\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371850 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-crio-socket\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371869 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-kubelet\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371895 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.371916 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-trusted-ca-bundle\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:54.372000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.372003 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ks47\" (UniqueName: \"kubernetes.io/projected/a619b2e9-c604-4179-aa69-aa822f209f0c-kube-api-access-9ks47\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:54.372637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.372079 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-kubernetes\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.372637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.372085 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2293549b-8f94-4bf2-8755-e83e4350fd79-iptables-alerter-script\") pod \"iptables-alerter-lzl7q\" (UID: \"2293549b-8f94-4bf2-8755-e83e4350fd79\") " pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:54.372637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.372108 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-cni-bin\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.372637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.372130 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-original-pull-secret\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:54.372637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.372181 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-cni-netd\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.372637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.372213 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-host\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.372637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.372229 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-etc-selinux\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.372637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.372266 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-data-volume\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:54.372637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.372286 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-trusted-ca-bundle\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:54.372637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.372354 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-data-volume\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:54.414865 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.414811 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 19:15:54.415118 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.415092 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:15:54.415370 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.415346 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-46fzl\"" Oct 30 19:15:54.415370 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.415366 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xzg6c\"" Oct 30 19:15:54.415506 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.415379 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-qjhd5\"" Oct 30 19:15:54.415506 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.415419 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6wdgf\"" Oct 30 19:15:54.415506 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.415420 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mnhjf\"" Oct 30 19:15:54.415506 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.415419 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:15:54.418640 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.418625 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 19:15:54.418640 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.418632 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 19:15:54.418752 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.418660 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 19:15:54.418895 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.418880 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:15:54.418947 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.418922 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 19:15:54.419007 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.418953 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:15:54.419007 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.418958 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:15:54.419007 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.418970 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:15:54.419101 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.418994 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:15:54.419129 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.419098 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/241808e4-15ac-452d-9139-85411bc375cd-cert\") pod \"ingress-canary-rp9mn\" (UID: \"241808e4-15ac-452d-9139-85411bc375cd\") " pod="openshift-ingress-canary/ingress-canary-rp9mn" Oct 30 19:15:54.421334 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.421318 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-signing-key\") pod \"service-ca-5fffc97bdf-m92f6\" (UID: \"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:54.421369 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.421318 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-signing-cabundle\") pod \"service-ca-5fffc97bdf-m92f6\" (UID: \"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:54.422414 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.422400 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-service-ca-bundle\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:54.422466 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.422424 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-daemon-config\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.427737 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.427718 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw5gw\" (UniqueName: \"kubernetes.io/projected/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-api-access-cw5gw\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:54.427737 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.427726 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq9wm\" (UniqueName: \"kubernetes.io/projected/d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0-kube-api-access-jq9wm\") pod \"migrator-66df5d496b-fs446\" (UID: \"d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fs446" Oct 30 19:15:54.428145 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.428128 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8gb9\" (UniqueName: \"kubernetes.io/projected/0d921aae-0b5b-4f3f-9ad5-33fcacb7213b-kube-api-access-k8gb9\") pod \"service-ca-5fffc97bdf-m92f6\" (UID: \"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" Oct 30 19:15:54.428276 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.428259 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6xm9\" (UniqueName: \"kubernetes.io/projected/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-kube-api-access-c6xm9\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:54.433959 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.433941 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ks47\" (UniqueName: \"kubernetes.io/projected/a619b2e9-c604-4179-aa69-aa822f209f0c-kube-api-access-9ks47\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:54.434408 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.434389 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdrqv\" (UniqueName: \"kubernetes.io/projected/bd91ed10-7ecf-415f-9906-3a79c23c8229-kube-api-access-wdrqv\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.434547 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.434527 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnzd6\" (UniqueName: \"kubernetes.io/projected/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-kube-api-access-qnzd6\") pod \"node-resolver-25zlr\" (UID: \"ddd92762-f7c4-4f85-80e1-0fe7045e78e8\") " pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:54.434584 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.434556 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v974s\" (UniqueName: \"kubernetes.io/projected/373b617b-0896-420c-b2fd-38635c62a21c-kube-api-access-v974s\") pod \"node-ca-s4k9z\" (UID: \"373b617b-0896-420c-b2fd-38635c62a21c\") " pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:54.434651 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.434633 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjmzh\" (UniqueName: \"kubernetes.io/projected/2293549b-8f94-4bf2-8755-e83e4350fd79-kube-api-access-jjmzh\") pod \"iptables-alerter-lzl7q\" (UID: \"2293549b-8f94-4bf2-8755-e83e4350fd79\") " pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:54.434687 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.434633 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hlgt\" (UniqueName: \"kubernetes.io/projected/a6f66f51-48bd-439d-92b7-a3c9dc33f591-kube-api-access-2hlgt\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.434822 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.434799 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6245n\" (UniqueName: \"kubernetes.io/projected/42b67c56-566b-4319-93ff-165d6cefdf19-kube-api-access-6245n\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.435068 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.435053 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzd7b\" (UniqueName: \"kubernetes.io/projected/241808e4-15ac-452d-9139-85411bc375cd-kube-api-access-qzd7b\") pod \"ingress-canary-rp9mn\" (UID: \"241808e4-15ac-452d-9139-85411bc375cd\") " pod="openshift-ingress-canary/ingress-canary-rp9mn" Oct 30 19:15:54.435815 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.435797 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrg7\" (UniqueName: \"kubernetes.io/projected/3d1a766f-5256-4044-98d5-f551f811e684-kube-api-access-fgrg7\") pod \"network-check-target-ql75b\" (UID: \"3d1a766f-5256-4044-98d5-f551f811e684\") " pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:54.436001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.435966 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7xhf\" (UniqueName: \"kubernetes.io/projected/dc918fe6-f902-419e-b4f1-04690e999f47-kube-api-access-q7xhf\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.436111 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.436092 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r5xm\" (UniqueName: \"kubernetes.io/projected/a62fb314-b483-4dcd-8170-a818757da247-kube-api-access-5r5xm\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.436258 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.436245 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh4p5\" (UniqueName: \"kubernetes.io/projected/44f7fda2-c7d3-470c-9080-1f931fac9baa-kube-api-access-jh4p5\") pod \"periodic-gathering-hjklm-r9jn7\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:15:54.472818 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.472799 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-modprobe-d\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.472870 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.472824 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-etc-kubernetes\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.472870 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.472842 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-crio-socket\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:54.472870 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.472855 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-kubelet\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.472990 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.472883 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-kubernetes\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.472990 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.472898 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-kubelet\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.472990 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.472908 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-cni-bin\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.472990 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.472941 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-crio-socket\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:54.472990 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.472950 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-modprobe-d\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.472990 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.472957 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-cni-netd\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.472990 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.472945 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-cni-bin\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.472973 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-host\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473011 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-etc-selinux\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473014 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-cni-netd\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473009 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-kubernetes\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473022 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-etc-kubernetes\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473042 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-cni-multus\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473064 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-etc-selinux\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473067 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2293549b-8f94-4bf2-8755-e83e4350fd79-host-slash\") pod \"iptables-alerter-lzl7q\" (UID: \"2293549b-8f94-4bf2-8755-e83e4350fd79\") " pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473078 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-cni-multus\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473084 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2293549b-8f94-4bf2-8755-e83e4350fd79-host-slash\") pod \"iptables-alerter-lzl7q\" (UID: \"2293549b-8f94-4bf2-8755-e83e4350fd79\") " pod="openshift-network-operator/iptables-alerter-lzl7q" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473045 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-host\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473094 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-hostroot\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473113 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-hostroot\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473139 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-slash\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473164 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473187 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473210 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysconfig\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473224 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-slash\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473233 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysctl-d\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473257 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-lib-modules\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473267 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysconfig\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473278 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-cnibin\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:54.473287 4433 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:15:54.473313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473302 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-run-netns\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473328 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-socket-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473351 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-system-cni-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473371 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-lib-modules\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473372 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysctl-d\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473376 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-multus-certs\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473414 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-system-cni-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473414 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-multus-certs\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473443 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-socket-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:54.473454 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls podName:7dcbf677-ca92-4306-bc45-ab9954ed7ac2 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:54.973414805 +0000 UTC m=+1.908202316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls") pod "dns-default-4j2jg" (UID: "7dcbf677-ca92-4306-bc45-ab9954ed7ac2") : secret "dns-default-metrics-tls" not found Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473377 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-run-netns\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473445 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-cnibin\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473476 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-log-socket\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473494 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-dbus\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473508 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-systemd\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473511 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-log-socket\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473528 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-registration-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473551 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-netns\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473563 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-dbus\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473573 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-var-lib-kubelet\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473564 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-systemd\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473598 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-system-cni-dir\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473603 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-registration-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473612 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-netns\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473622 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-conf-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473630 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-var-lib-kubelet\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473632 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473648 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-system-cni-dir\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473654 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-conf-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473648 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473685 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-kubelet-config\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473686 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473705 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-device-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473726 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-run\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473735 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-device-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473739 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/373b617b-0896-420c-b2fd-38635c62a21c-host\") pod \"node-ca-s4k9z\" (UID: \"373b617b-0896-420c-b2fd-38635c62a21c\") " pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473738 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed1dc05f-2a13-4dd2-9d20-f5e96f17099f-kubelet-config\") pod \"global-pull-secret-syncer-6vks7\" (UID: \"ed1dc05f-2a13-4dd2-9d20-f5e96f17099f\") " pod="kube-system/global-pull-secret-syncer-6vks7" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473758 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-cnibin\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473761 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-run\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473772 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-ovn\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473787 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-systemd-units\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473765 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/373b617b-0896-420c-b2fd-38635c62a21c-host\") pod \"node-ca-s4k9z\" (UID: \"373b617b-0896-420c-b2fd-38635c62a21c\") " pod="openshift-image-registry/node-ca-s4k9z" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473801 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-ovn\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473802 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473813 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-systemd-units\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473788 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-cnibin\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473828 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:54.473846 4433 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473863 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-systemd\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:54.473883 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs podName:a619b2e9-c604-4179-aa69-aa822f209f0c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:54.973872107 +0000 UTC m=+1.908659609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs") pod "network-metrics-daemon-7gj92" (UID: "a619b2e9-c604-4179-aa69-aa822f209f0c") : secret "metrics-daemon-secret" not found Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473847 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-run-systemd\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473902 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-etc-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473908 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc918fe6-f902-419e-b4f1-04690e999f47-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2hmn\" (UID: \"dc918fe6-f902-419e-b4f1-04690e999f47\") " pod="openshift-multus/multus-additional-cni-plugins-v2hmn" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473918 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473932 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-etc-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473935 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysctl-conf\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473948 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-sys-fs\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473954 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473963 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.473993 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-cni-bin\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474011 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-var-lib-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474027 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-cni-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474034 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-var-lib-cni-bin\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474039 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-hosts-file\") pod \"node-resolver-25zlr\" (UID: \"ddd92762-f7c4-4f85-80e1-0fe7045e78e8\") " pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474010 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-sys-fs\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:54.474049 4433 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474056 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-socket-dir-parent\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474071 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-k8s-cni-cncf-io\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474072 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-var-lib-openvswitch\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474082 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-etc-sysctl-conf\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:54.474091 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls podName:69a39e1d-7ab3-40a0-9f5d-ae469a5d326c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:54.974076711 +0000 UTC m=+1.908864233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjh7f" (UID: "69a39e1d-7ab3-40a0-9f5d-ae469a5d326c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474099 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-host-run-k8s-cni-cncf-io\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474106 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddd92762-f7c4-4f85-80e1-0fe7045e78e8-hosts-file\") pod \"node-resolver-25zlr\" (UID: \"ddd92762-f7c4-4f85-80e1-0fe7045e78e8\") " pod="openshift-dns/node-resolver-25zlr" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474113 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-node-log\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474124 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-cni-dir\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474129 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-node-log\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474134 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6f66f51-48bd-439d-92b7-a3c9dc33f591-multus-socket-dir-parent\") pod \"multus-kldq6\" (UID: \"a6f66f51-48bd-439d-92b7-a3c9dc33f591\") " pod="openshift-multus/multus-kldq6" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474145 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474169 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-sys\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.474148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474189 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.476195 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474189 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.476195 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474215 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-kubelet\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.476195 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474230 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd91ed10-7ecf-415f-9906-3a79c23c8229-sys\") pod \"tuned-8ch84\" (UID: \"bd91ed10-7ecf-415f-9906-3a79c23c8229\") " pod="openshift-cluster-node-tuning-operator/tuned-8ch84" Oct 30 19:15:54.476195 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474230 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42b67c56-566b-4319-93ff-165d6cefdf19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4cx2w\" (UID: \"42b67c56-566b-4319-93ff-165d6cefdf19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4cx2w" Oct 30 19:15:54.476195 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.474266 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a62fb314-b483-4dcd-8170-a818757da247-host-kubelet\") pod \"ovnkube-node-kg9n8\" (UID: \"a62fb314-b483-4dcd-8170-a818757da247\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:54.564641 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.564580 4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:15:54.689388 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.689328 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:54.977695 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.977632 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:54.977695 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.977671 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:54.977847 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:54.977706 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:54.977847 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:54.977759 4433 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:15:54.977847 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:54.977783 4433 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:15:54.977847 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:54.977793 4433 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:15:54.977847 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:54.977823 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs podName:a619b2e9-c604-4179-aa69-aa822f209f0c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:55.977809524 +0000 UTC m=+2.912597027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs") pod "network-metrics-daemon-7gj92" (UID: "a619b2e9-c604-4179-aa69-aa822f209f0c") : secret "metrics-daemon-secret" not found Oct 30 19:15:54.977847 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:54.977835 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls podName:7dcbf677-ca92-4306-bc45-ab9954ed7ac2 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:55.977829683 +0000 UTC m=+2.912617185 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls") pod "dns-default-4j2jg" (UID: "7dcbf677-ca92-4306-bc45-ab9954ed7ac2") : secret "dns-default-metrics-tls" not found Oct 30 19:15:54.977847 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:54.977845 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls podName:69a39e1d-7ab3-40a0-9f5d-ae469a5d326c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:55.977840284 +0000 UTC m=+2.912627787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjh7f" (UID: "69a39e1d-7ab3-40a0-9f5d-ae469a5d326c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:15:55.979709 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:55.979665 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:55.982666 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:55.982631 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:55.982811 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:55.982702 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:55.982811 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:55.982735 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:55.982811 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:55.982748 4433 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:15:55.983084 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:55.982817 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls podName:7dcbf677-ca92-4306-bc45-ab9954ed7ac2 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.982798798 +0000 UTC m=+4.917586305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls") pod "dns-default-4j2jg" (UID: "7dcbf677-ca92-4306-bc45-ab9954ed7ac2") : secret "dns-default-metrics-tls" not found Oct 30 19:15:55.983084 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:55.982843 4433 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:15:55.983084 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:55.982850 4433 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:15:55.983084 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:55.982897 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs podName:a619b2e9-c604-4179-aa69-aa822f209f0c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.982883985 +0000 UTC m=+4.917671489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs") pod "network-metrics-daemon-7gj92" (UID: "a619b2e9-c604-4179-aa69-aa822f209f0c") : secret "metrics-daemon-secret" not found Oct 30 19:15:55.983084 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:55.982912 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls podName:69a39e1d-7ab3-40a0-9f5d-ae469a5d326c nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.982905848 +0000 UTC m=+4.917693350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjh7f" (UID: "69a39e1d-7ab3-40a0-9f5d-ae469a5d326c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:15:56.263012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:56.262917 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hjklm-r9jn7" event={"ID":"44f7fda2-c7d3-470c-9080-1f931fac9baa","Type":"ContainerStarted","Data":"8f5d4eeb4c8d73f1db1d35976741926823d65ca1b9ac898a4179fe59cf9ac202"} Oct 30 19:15:57.661829 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:57.661796 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:15:57.998286 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:57.998195 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:15:57.998286 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:57.998256 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:15:57.998505 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:57.998291 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:15:57.998505 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:57.998377 4433 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:15:57.998505 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:57.998396 4433 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:15:57.998505 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:57.998380 4433 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:15:57.998505 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:57.998448 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs podName:a619b2e9-c604-4179-aa69-aa822f209f0c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.998434866 +0000 UTC m=+8.933222369 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs") pod "network-metrics-daemon-7gj92" (UID: "a619b2e9-c604-4179-aa69-aa822f209f0c") : secret "metrics-daemon-secret" not found Oct 30 19:15:57.998505 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:57.998465 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls podName:7dcbf677-ca92-4306-bc45-ab9954ed7ac2 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.998455864 +0000 UTC m=+8.933243366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls") pod "dns-default-4j2jg" (UID: "7dcbf677-ca92-4306-bc45-ab9954ed7ac2") : secret "dns-default-metrics-tls" not found Oct 30 19:15:57.998505 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:15:57.998478 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls podName:69a39e1d-7ab3-40a0-9f5d-ae469a5d326c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.998470803 +0000 UTC m=+8.933258305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjh7f" (UID: "69a39e1d-7ab3-40a0-9f5d-ae469a5d326c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:15:59.271817 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:59.271783 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-m92f6" event={"ID":"0d921aae-0b5b-4f3f-9ad5-33fcacb7213b","Type":"ContainerStarted","Data":"95291af65efaef2e4f84a1c6f6332d6a771b0898bb8f85340bac79a534640cd3"} Oct 30 19:15:59.292190 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:59.292162 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:15:59.584697 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:59.584668 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-25zlr_ddd92762-f7c4-4f85-80e1-0fe7045e78e8/dns-node-resolver/0.log" Oct 30 19:15:59.979637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:15:59.979547 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:00.291914 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:00.291829 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:00.523519 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:00.523479 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:00.754110 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:00.754078 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:00.984429 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:00.984404 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s4k9z_373b617b-0896-420c-b2fd-38635c62a21c/node-ca/0.log" Oct 30 19:16:01.621899 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:01.621867 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:01.795109 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:01.795077 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rp9mn_241808e4-15ac-452d-9139-85411bc375cd/serve-healthcheck-canary/0.log" Oct 30 19:16:02.023577 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:02.023500 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:16:02.023577 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:02.023542 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:16:02.023804 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:02.023596 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:16:02.023804 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:16:02.023651 4433 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:16:02.023804 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:16:02.023714 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs podName:a619b2e9-c604-4179-aa69-aa822f209f0c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:10.023697102 +0000 UTC m=+16.958484624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs") pod "network-metrics-daemon-7gj92" (UID: "a619b2e9-c604-4179-aa69-aa822f209f0c") : secret "metrics-daemon-secret" not found Oct 30 19:16:02.023804 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:16:02.023725 4433 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:16:02.023804 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:16:02.023786 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls podName:69a39e1d-7ab3-40a0-9f5d-ae469a5d326c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:10.023767877 +0000 UTC m=+16.958555396 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjh7f" (UID: "69a39e1d-7ab3-40a0-9f5d-ae469a5d326c") : secret "insights-runtime-extractor-tls" not found Oct 30 19:16:02.024115 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:02.024092 4433 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:16:02.026074 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:02.026049 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7dcbf677-ca92-4306-bc45-ab9954ed7ac2-metrics-tls\") pod \"dns-default-4j2jg\" (UID: \"7dcbf677-ca92-4306-bc45-ab9954ed7ac2\") " pod="openshift-dns/dns-default-4j2jg" Oct 30 19:16:02.029039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:02.029017 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:02.062843 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:02.062814 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4j2jg" Oct 30 19:16:02.185825 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:02.185795 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-fs446_d7f1e0ce-e2d3-4838-bd81-7138a77b5aa0/migrator/0.log" Oct 30 19:16:02.251430 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:02.251401 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:02.430487 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:02.430390 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4j2jg"] Oct 30 19:16:02.436762 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:16:02.436732 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dcbf677_ca92_4306_bc45_ab9954ed7ac2.slice/crio-24de2247b269b79f5cf02b2900919c7cda10f8853be195751a6e4cd1d028c7df WatchSource:0}: Error finding container 24de2247b269b79f5cf02b2900919c7cda10f8853be195751a6e4cd1d028c7df: Status 404 returned error can't find the container with id 24de2247b269b79f5cf02b2900919c7cda10f8853be195751a6e4cd1d028c7df Oct 30 19:16:02.480108 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:02.480041 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:02.827939 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:02.827890 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:03.282073 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:03.281964 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4j2jg" event={"ID":"7dcbf677-ca92-4306-bc45-ab9954ed7ac2","Type":"ContainerStarted","Data":"24de2247b269b79f5cf02b2900919c7cda10f8853be195751a6e4cd1d028c7df"} Oct 30 19:16:03.622120 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:03.622082 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:03.942890 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:03.942809 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:16:03.943331 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:03.943141 4433 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:16:04.030461 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:04.030426 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:04.258269 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:04.258179 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:04.486748 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:04.486722 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:04.724638 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:04.724607 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:04.970134 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:04.970104 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:05.292973 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:05.292896 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4j2jg" event={"ID":"7dcbf677-ca92-4306-bc45-ab9954ed7ac2","Type":"ContainerStarted","Data":"2b22250db014806c4309b5d50c40af37323af45c170412a1c5685a14c7569ba2"} Oct 30 19:16:05.293127 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:05.292999 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4j2jg" event={"ID":"7dcbf677-ca92-4306-bc45-ab9954ed7ac2","Type":"ContainerStarted","Data":"27dcdf2128a1f9a1457af4098b49da4d924617571602b9ab12fad3235815ad48"} Oct 30 19:16:05.293127 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:05.293053 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4j2jg" Oct 30 19:16:05.428476 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:05.428445 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:05.666514 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:05.666483 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:06.035103 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:06.035027 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:06.280515 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:06.280484 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:06.556736 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:06.556698 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:06.842043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:06.842017 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:07.631843 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:07.631813 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:08.037243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:08.037166 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:08.271933 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:08.271899 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:08.510667 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:08.510637 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:08.919389 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:08.919357 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:09.638398 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:09.638365 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:09.871533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:09.871501 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:10.076198 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.076163 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:16:10.076604 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.076258 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:16:10.078675 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.078653 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69a39e1d-7ab3-40a0-9f5d-ae469a5d326c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjh7f\" (UID: \"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c\") " pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:16:10.078796 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.078739 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a619b2e9-c604-4179-aa69-aa822f209f0c-metrics-certs\") pod \"network-metrics-daemon-7gj92\" (UID: \"a619b2e9-c604-4179-aa69-aa822f209f0c\") " pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:16:10.118535 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.118493 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:10.162903 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.162865 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7gj92" Oct 30 19:16:10.163968 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.163948 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hjh7f" Oct 30 19:16:10.365869 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.365782 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:10.534295 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.534264 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7gj92"] Oct 30 19:16:10.577302 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:16:10.577271 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda619b2e9_c604_4179_aa69_aa822f209f0c.slice/crio-745c4b9066607bd7f547da159c2d132680f40b697abb24007215bc22b5d1e967 WatchSource:0}: Error finding container 745c4b9066607bd7f547da159c2d132680f40b697abb24007215bc22b5d1e967: Status 404 returned error can't find the container with id 745c4b9066607bd7f547da159c2d132680f40b697abb24007215bc22b5d1e967 Oct 30 19:16:10.589711 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.589688 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hjh7f"] Oct 30 19:16:10.592838 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:16:10.592816 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a39e1d_7ab3_40a0_9f5d_ae469a5d326c.slice/crio-5d75b59b26116ffb3da1efd11304a3c2bffdedbaba9aaf54d471805b8e90a87c WatchSource:0}: Error finding container 5d75b59b26116ffb3da1efd11304a3c2bffdedbaba9aaf54d471805b8e90a87c: Status 404 returned error can't find the container with id 5d75b59b26116ffb3da1efd11304a3c2bffdedbaba9aaf54d471805b8e90a87c Oct 30 19:16:10.601099 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.601077 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:10.894356 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.894246 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f57c7f55b-mpdsx"] Oct 30 19:16:10.896269 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.896244 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:10.898211 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.898186 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:10.949328 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.949292 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2q27v\"" Oct 30 19:16:10.949526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.949489 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 19:16:10.949634 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.949620 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 19:16:10.949746 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.949735 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 19:16:10.983091 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.983052 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e80568cf-d1b8-4049-a5c8-6975b76386f0-image-registry-private-configuration\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:10.983091 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.983093 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e80568cf-d1b8-4049-a5c8-6975b76386f0-registry-certificates\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:10.983360 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.983129 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q252\" (UniqueName: \"kubernetes.io/projected/e80568cf-d1b8-4049-a5c8-6975b76386f0-kube-api-access-7q252\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:10.983360 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.983215 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e80568cf-d1b8-4049-a5c8-6975b76386f0-installation-pull-secrets\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:10.983360 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.983242 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e80568cf-d1b8-4049-a5c8-6975b76386f0-registry-tls\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:10.983360 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.983263 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e80568cf-d1b8-4049-a5c8-6975b76386f0-ca-trust-extracted\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:10.983360 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.983286 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e80568cf-d1b8-4049-a5c8-6975b76386f0-trusted-ca\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:10.983360 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.983360 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e80568cf-d1b8-4049-a5c8-6975b76386f0-bound-sa-token\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:10.999514 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:10.999486 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f57c7f55b-mpdsx"] Oct 30 19:16:11.049173 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.049140 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 19:16:11.083842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.083808 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e80568cf-d1b8-4049-a5c8-6975b76386f0-image-registry-private-configuration\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.084298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.083853 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e80568cf-d1b8-4049-a5c8-6975b76386f0-registry-certificates\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.084298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.083888 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q252\" (UniqueName: \"kubernetes.io/projected/e80568cf-d1b8-4049-a5c8-6975b76386f0-kube-api-access-7q252\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.084298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.083931 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e80568cf-d1b8-4049-a5c8-6975b76386f0-installation-pull-secrets\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.084298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.083958 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e80568cf-d1b8-4049-a5c8-6975b76386f0-registry-tls\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.084298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.084003 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e80568cf-d1b8-4049-a5c8-6975b76386f0-ca-trust-extracted\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.084298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.084038 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e80568cf-d1b8-4049-a5c8-6975b76386f0-trusted-ca\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.084298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.084068 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e80568cf-d1b8-4049-a5c8-6975b76386f0-bound-sa-token\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.084840 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.084819 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e80568cf-d1b8-4049-a5c8-6975b76386f0-ca-trust-extracted\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.085259 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.085096 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e80568cf-d1b8-4049-a5c8-6975b76386f0-registry-certificates\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.085546 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.085526 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e80568cf-d1b8-4049-a5c8-6975b76386f0-trusted-ca\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.087220 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.087195 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e80568cf-d1b8-4049-a5c8-6975b76386f0-installation-pull-secrets\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.087303 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.087290 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e80568cf-d1b8-4049-a5c8-6975b76386f0-registry-tls\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.087347 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.087318 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e80568cf-d1b8-4049-a5c8-6975b76386f0-image-registry-private-configuration\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.141770 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.141739 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q252\" (UniqueName: \"kubernetes.io/projected/e80568cf-d1b8-4049-a5c8-6975b76386f0-kube-api-access-7q252\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.142233 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.142215 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e80568cf-d1b8-4049-a5c8-6975b76386f0-bound-sa-token\") pod \"image-registry-f57c7f55b-mpdsx\" (UID: \"e80568cf-d1b8-4049-a5c8-6975b76386f0\") " pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.170321 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.170247 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:11.207520 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.207491 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:11.312612 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.312559 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjh7f" event={"ID":"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c","Type":"ContainerStarted","Data":"890ff0fd84b6f9c9c7c8def4d8bf9a90aea70c16b2802f43597edc31a538e7df"} Oct 30 19:16:11.312782 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.312638 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjh7f" event={"ID":"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c","Type":"ContainerStarted","Data":"5d75b59b26116ffb3da1efd11304a3c2bffdedbaba9aaf54d471805b8e90a87c"} Oct 30 19:16:11.313653 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.313631 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7gj92" event={"ID":"a619b2e9-c604-4179-aa69-aa822f209f0c","Type":"ContainerStarted","Data":"745c4b9066607bd7f547da159c2d132680f40b697abb24007215bc22b5d1e967"} Oct 30 19:16:11.434235 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.434150 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:11.627517 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:11.627483 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f57c7f55b-mpdsx"] Oct 30 19:16:11.632319 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:16:11.632292 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80568cf_d1b8_4049_a5c8_6975b76386f0.slice/crio-d0040a345773b4756d80fc58633fa95786ddcedfcf769f945c7d97f1c7fa82c8 WatchSource:0}: Error finding container d0040a345773b4756d80fc58633fa95786ddcedfcf769f945c7d97f1c7fa82c8: Status 404 returned error can't find the container with id d0040a345773b4756d80fc58633fa95786ddcedfcf769f945c7d97f1c7fa82c8 Oct 30 19:16:12.318138 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:12.318106 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjh7f" event={"ID":"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c","Type":"ContainerStarted","Data":"d7a5585079d9e8d362bd321abfe68ba0679f336c51a0b22588bf012bfa39ac93"} Oct 30 19:16:12.319851 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:12.319826 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7gj92" event={"ID":"a619b2e9-c604-4179-aa69-aa822f209f0c","Type":"ContainerStarted","Data":"a14acb2a9bff9200f3b8369ba4adb68e68241714749a80d4e77ed9d2b98cf401"} Oct 30 19:16:12.320008 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:12.319868 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7gj92" event={"ID":"a619b2e9-c604-4179-aa69-aa822f209f0c","Type":"ContainerStarted","Data":"61c29fdd36ae683c02a69e44ff2a4c47cffd3ca90935cf30840f59876567e762"} Oct 30 19:16:12.323745 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:12.323722 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" event={"ID":"e80568cf-d1b8-4049-a5c8-6975b76386f0","Type":"ContainerStarted","Data":"553f71df2bca5c36e45a9703818ba96bcbad5fc5e0fb303284dc6bd6910089eb"} Oct 30 19:16:12.323873 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:12.323754 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" event={"ID":"e80568cf-d1b8-4049-a5c8-6975b76386f0","Type":"ContainerStarted","Data":"d0040a345773b4756d80fc58633fa95786ddcedfcf769f945c7d97f1c7fa82c8"} Oct 30 19:16:12.323873 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:12.323845 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:12.558975 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:12.558916 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" podStartSLOduration=2.558903312 podStartE2EDuration="2.558903312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:12.558279594 +0000 UTC m=+19.493067131" watchObservedRunningTime="2026-10-30 19:16:12.558903312 +0000 UTC m=+19.493690836" Oct 30 19:16:13.779377 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:13.779328 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:14.017912 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:14.017826 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:14.263793 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:14.263763 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:14.332833 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:14.332799 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjh7f" event={"ID":"69a39e1d-7ab3-40a0-9f5d-ae469a5d326c","Type":"ContainerStarted","Data":"72edb3653647ebacade96f58d56e473bb270a80bfffe018a6797124e4566418e"} Oct 30 19:16:14.869632 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:14.869596 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2sd6"] Oct 30 19:16:14.871618 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:14.871595 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2sd6" Oct 30 19:16:14.884347 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:14.884319 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:14.910262 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:14.910232 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/458568b4-381b-46b0-9943-874c5b6bd8c4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-h2sd6\" (UID: \"458568b4-381b-46b0-9943-874c5b6bd8c4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2sd6" Oct 30 19:16:14.923808 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:14.923787 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-48qsp\"" Oct 30 19:16:14.923869 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:14.923853 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 19:16:14.935280 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:14.935249 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2sd6"] Oct 30 19:16:15.011392 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:15.011349 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/458568b4-381b-46b0-9943-874c5b6bd8c4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-h2sd6\" (UID: \"458568b4-381b-46b0-9943-874c5b6bd8c4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2sd6" Oct 30 19:16:15.013727 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:15.013703 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/458568b4-381b-46b0-9943-874c5b6bd8c4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-h2sd6\" (UID: \"458568b4-381b-46b0-9943-874c5b6bd8c4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2sd6" Oct 30 19:16:15.148919 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:15.148844 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:15.180968 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:15.180935 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2sd6" Oct 30 19:16:15.299088 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:15.299052 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4j2jg" Oct 30 19:16:15.556523 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:15.556391 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2sd6"] Oct 30 19:16:15.561838 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:16:15.561810 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458568b4_381b_46b0_9943_874c5b6bd8c4.slice/crio-662f3baf4d582b59f1cbd4a68ea32974df9239d9eafbad5a0693ad4624e8516d WatchSource:0}: Error finding container 662f3baf4d582b59f1cbd4a68ea32974df9239d9eafbad5a0693ad4624e8516d: Status 404 returned error can't find the container with id 662f3baf4d582b59f1cbd4a68ea32974df9239d9eafbad5a0693ad4624e8516d Oct 30 19:16:16.339922 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:16.339887 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2sd6" event={"ID":"458568b4-381b-46b0-9943-874c5b6bd8c4","Type":"ContainerStarted","Data":"662f3baf4d582b59f1cbd4a68ea32974df9239d9eafbad5a0693ad4624e8516d"} Oct 30 19:16:16.941303 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:16.941273 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg9n8" Oct 30 19:16:17.344427 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:17.344391 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2sd6" event={"ID":"458568b4-381b-46b0-9943-874c5b6bd8c4","Type":"ContainerStarted","Data":"6564a660b054089e321f1b52e266c4aca28f21721b4966a2c751eb8d0c830701"} Oct 30 19:16:17.344922 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:17.344603 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2sd6" Oct 30 19:16:17.349085 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:17.349061 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2sd6" Oct 30 19:16:17.457946 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:17.457880 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h2sd6" podStartSLOduration=2.068857881 podStartE2EDuration="3.45786676s" totalImagesPullingTime="1.389008879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:17.457640249 +0000 UTC m=+24.392427776" watchObservedRunningTime="2026-10-30 19:16:17.45786676 +0000 UTC m=+24.392654285" Oct 30 19:16:17.959726 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:17.959696 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lv4mc" Oct 30 19:16:18.115042 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:18.115015 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:18.343276 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:18.343246 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:18.575813 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:18.575782 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:18.812705 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:18.812674 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:23.356571 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:23.356534 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:23.367403 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:16:23.367365 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:16:23.629134 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:23.629029 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:23.882937 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:23.882846 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:23.969914 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:23.969887 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bsfgw"] Oct 30 19:16:23.994164 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:23.994132 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.072474 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.072438 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:16:24.073630 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.073608 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06e8ca20-b81c-4079-8707-7028977e19f6-node-exporter-tls\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.073725 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.073650 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06e8ca20-b81c-4079-8707-7028977e19f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.073725 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.073684 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06e8ca20-b81c-4079-8707-7028977e19f6-metrics-client-ca\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.073800 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.073774 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjll9\" (UniqueName: \"kubernetes.io/projected/06e8ca20-b81c-4079-8707-7028977e19f6-kube-api-access-pjll9\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.073845 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.073812 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06e8ca20-b81c-4079-8707-7028977e19f6-root\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.073882 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.073858 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06e8ca20-b81c-4079-8707-7028977e19f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.073927 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.073904 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06e8ca20-b81c-4079-8707-7028977e19f6-sys\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.074012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.073993 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06e8ca20-b81c-4079-8707-7028977e19f6-node-exporter-wtmp\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.074073 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.074033 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06e8ca20-b81c-4079-8707-7028977e19f6-node-exporter-textfile\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.095907 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.095870 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:16:24.095907 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.095875 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2dwz4\"" Oct 30 19:16:24.122894 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.122857 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:16:24.126354 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.126330 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:16:24.145540 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.145449 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:16:24.145540 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.145502 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:16:24.148480 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.148432 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:24.174691 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.174650 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06e8ca20-b81c-4079-8707-7028977e19f6-root\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.174691 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.174685 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06e8ca20-b81c-4079-8707-7028977e19f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.174884 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.174705 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06e8ca20-b81c-4079-8707-7028977e19f6-sys\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.174884 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.174725 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06e8ca20-b81c-4079-8707-7028977e19f6-node-exporter-wtmp\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.174884 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.174754 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06e8ca20-b81c-4079-8707-7028977e19f6-node-exporter-textfile\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.174884 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.174762 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06e8ca20-b81c-4079-8707-7028977e19f6-root\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.174884 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.174809 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06e8ca20-b81c-4079-8707-7028977e19f6-sys\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.174884 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.174855 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06e8ca20-b81c-4079-8707-7028977e19f6-node-exporter-tls\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.175172 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.174896 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06e8ca20-b81c-4079-8707-7028977e19f6-node-exporter-wtmp\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.175172 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.174907 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06e8ca20-b81c-4079-8707-7028977e19f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.175172 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.174952 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06e8ca20-b81c-4079-8707-7028977e19f6-metrics-client-ca\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.175172 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.175002 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjll9\" (UniqueName: \"kubernetes.io/projected/06e8ca20-b81c-4079-8707-7028977e19f6-kube-api-access-pjll9\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.175379 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.175187 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06e8ca20-b81c-4079-8707-7028977e19f6-node-exporter-textfile\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.175379 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.175279 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06e8ca20-b81c-4079-8707-7028977e19f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.175466 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.175446 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06e8ca20-b81c-4079-8707-7028977e19f6-metrics-client-ca\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.177309 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.177287 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06e8ca20-b81c-4079-8707-7028977e19f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.177412 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.177308 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06e8ca20-b81c-4079-8707-7028977e19f6-node-exporter-tls\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.232278 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.232237 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjll9\" (UniqueName: \"kubernetes.io/projected/06e8ca20-b81c-4079-8707-7028977e19f6-kube-api-access-pjll9\") pod \"node-exporter-bsfgw\" (UID: \"06e8ca20-b81c-4079-8707-7028977e19f6\") " pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.343531 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.343491 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bsfgw" Oct 30 19:16:24.383554 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:16:24.383524 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e8ca20_b81c_4079_8707_7028977e19f6.slice/crio-79b49388ff4a201488a0ade7c8b6a265383b3c9a26cbd8eb06b325f2fce94c6a WatchSource:0}: Error finding container 79b49388ff4a201488a0ade7c8b6a265383b3c9a26cbd8eb06b325f2fce94c6a: Status 404 returned error can't find the container with id 79b49388ff4a201488a0ade7c8b6a265383b3c9a26cbd8eb06b325f2fce94c6a Oct 30 19:16:24.387612 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.387583 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:24.620814 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:24.620779 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:25.369565 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:25.369526 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bsfgw" event={"ID":"06e8ca20-b81c-4079-8707-7028977e19f6","Type":"ContainerStarted","Data":"79b49388ff4a201488a0ade7c8b6a265383b3c9a26cbd8eb06b325f2fce94c6a"} Oct 30 19:16:25.430841 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:25.430807 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:25.650582 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:25.650485 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:26.339768 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.339733 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:26.631956 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.631929 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6f48b5bf56-wq96p"] Oct 30 19:16:26.647829 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.647795 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.648359 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.648333 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f48b5bf56-wq96p"] Oct 30 19:16:26.691169 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.691138 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-grpc-tls\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.691169 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.691172 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.691418 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.691194 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c142a503-11c1-4f87-aebb-f68610a7d227-metrics-client-ca\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.691418 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.691211 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-thanos-querier-tls\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.691418 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.691385 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.691548 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.691434 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.691548 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.691505 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c142a503-11c1-4f87-aebb-f68610a7d227-tmp\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.691623 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.691582 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxkxk\" (UniqueName: \"kubernetes.io/projected/c142a503-11c1-4f87-aebb-f68610a7d227-kube-api-access-zxkxk\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.691623 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.691606 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.699848 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.699793 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 19:16:26.700090 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.699915 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9cgftghtl6782\"" Oct 30 19:16:26.700090 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.699934 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 19:16:26.734430 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.734398 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 19:16:26.734430 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.734425 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lhcsh\"" Oct 30 19:16:26.792288 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.792245 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxkxk\" (UniqueName: \"kubernetes.io/projected/c142a503-11c1-4f87-aebb-f68610a7d227-kube-api-access-zxkxk\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.792288 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.792290 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.792512 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.792410 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-grpc-tls\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.792512 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.792448 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.792512 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.792471 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c142a503-11c1-4f87-aebb-f68610a7d227-metrics-client-ca\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.792512 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.792491 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-thanos-querier-tls\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.792694 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.792520 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.792694 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.792555 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.792694 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.792581 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c142a503-11c1-4f87-aebb-f68610a7d227-tmp\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.793035 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.792975 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c142a503-11c1-4f87-aebb-f68610a7d227-tmp\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.793319 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.793296 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c142a503-11c1-4f87-aebb-f68610a7d227-metrics-client-ca\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.795049 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.795024 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-thanos-querier-tls\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.795294 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.795274 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-grpc-tls\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.795336 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.795308 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.795336 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.795323 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.800605 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.800582 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 19:16:26.800701 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.800582 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 19:16:26.804896 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.804865 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.804999 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.804894 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c142a503-11c1-4f87-aebb-f68610a7d227-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:26.850299 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:26.850268 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxkxk\" (UniqueName: \"kubernetes.io/projected/c142a503-11c1-4f87-aebb-f68610a7d227-kube-api-access-zxkxk\") pod \"thanos-querier-6f48b5bf56-wq96p\" (UID: \"c142a503-11c1-4f87-aebb-f68610a7d227\") " pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:27.037009 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:27.036895 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:27.376759 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:27.376724 4433 generic.go:356] "Generic (PLEG): container finished" podID="06e8ca20-b81c-4079-8707-7028977e19f6" containerID="18940fe00c25f8c3b82ed5f7c5a0f1f29e8e393874fd67609e37507c5736017b" exitCode=0 Oct 30 19:16:27.376953 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:27.376768 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bsfgw" event={"ID":"06e8ca20-b81c-4079-8707-7028977e19f6","Type":"ContainerDied","Data":"18940fe00c25f8c3b82ed5f7c5a0f1f29e8e393874fd67609e37507c5736017b"} Oct 30 19:16:27.572667 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:27.572636 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f48b5bf56-wq96p"] Oct 30 19:16:27.578274 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:16:27.578233 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc142a503_11c1_4f87_aebb_f68610a7d227.slice/crio-3967589a7cdd7a61162e0978f934302bb6f80d71874ea832f92984e7236beeb6 WatchSource:0}: Error finding container 3967589a7cdd7a61162e0978f934302bb6f80d71874ea832f92984e7236beeb6: Status 404 returned error can't find the container with id 3967589a7cdd7a61162e0978f934302bb6f80d71874ea832f92984e7236beeb6 Oct 30 19:16:27.664873 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:27.664790 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ql75b" Oct 30 19:16:28.310867 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.310833 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:28.382455 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.382418 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bsfgw" event={"ID":"06e8ca20-b81c-4079-8707-7028977e19f6","Type":"ContainerStarted","Data":"fce9030289a1af6a6ae3d7876484fb666029830b9e5a675d0ab2bc042dbeafcc"} Oct 30 19:16:28.382633 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.382472 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bsfgw" event={"ID":"06e8ca20-b81c-4079-8707-7028977e19f6","Type":"ContainerStarted","Data":"3998377ebc164404f02f6b8855a8bf57ca4386e18a745584dc5a7f22ceaefb95"} Oct 30 19:16:28.383664 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.383635 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" event={"ID":"c142a503-11c1-4f87-aebb-f68610a7d227","Type":"ContainerStarted","Data":"3967589a7cdd7a61162e0978f934302bb6f80d71874ea832f92984e7236beeb6"} Oct 30 19:16:28.562509 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.562466 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:28.733187 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.733119 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bsfgw" podStartSLOduration=3.496561074 podStartE2EDuration="5.73309904s" totalImagesPullingTime="2.162713057s" totalInitContainerRuntime="73.824909ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:28.508422188 +0000 UTC m=+35.443209776" watchObservedRunningTime="2026-10-30 19:16:28.73309904 +0000 UTC m=+35.667886569" Oct 30 19:16:28.733680 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.733329 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-qpqdl"] Oct 30 19:16:28.750716 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.750684 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qpqdl" Oct 30 19:16:28.801029 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.800990 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-qpqdl"] Oct 30 19:16:28.811359 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.811310 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92dfr\" (UniqueName: \"kubernetes.io/projected/4d9fba76-deca-4eac-908e-03103b2f779e-kube-api-access-92dfr\") pod \"monitoring-plugin-77f4f76594-qpqdl\" (UID: \"4d9fba76-deca-4eac-908e-03103b2f779e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qpqdl" Oct 30 19:16:28.811534 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.811376 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4d9fba76-deca-4eac-908e-03103b2f779e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-qpqdl\" (UID: \"4d9fba76-deca-4eac-908e-03103b2f779e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qpqdl" Oct 30 19:16:28.814243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.814165 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:28.844565 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.844536 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 19:16:28.844565 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.844552 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7bcp4\"" Oct 30 19:16:28.912832 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.912794 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92dfr\" (UniqueName: \"kubernetes.io/projected/4d9fba76-deca-4eac-908e-03103b2f779e-kube-api-access-92dfr\") pod \"monitoring-plugin-77f4f76594-qpqdl\" (UID: \"4d9fba76-deca-4eac-908e-03103b2f779e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qpqdl" Oct 30 19:16:28.913030 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.912846 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4d9fba76-deca-4eac-908e-03103b2f779e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-qpqdl\" (UID: \"4d9fba76-deca-4eac-908e-03103b2f779e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qpqdl" Oct 30 19:16:28.915376 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.915349 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4d9fba76-deca-4eac-908e-03103b2f779e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-qpqdl\" (UID: \"4d9fba76-deca-4eac-908e-03103b2f779e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qpqdl" Oct 30 19:16:28.969938 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:28.969900 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92dfr\" (UniqueName: \"kubernetes.io/projected/4d9fba76-deca-4eac-908e-03103b2f779e-kube-api-access-92dfr\") pod \"monitoring-plugin-77f4f76594-qpqdl\" (UID: \"4d9fba76-deca-4eac-908e-03103b2f779e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qpqdl" Oct 30 19:16:29.074742 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:29.074652 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qpqdl" Oct 30 19:16:29.134834 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:29.134802 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:29.411603 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:29.411520 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:29.479358 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:29.479322 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-qpqdl"] Oct 30 19:16:29.483039 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:16:29.483007 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9fba76_deca_4eac_908e_03103b2f779e.slice/crio-ad31c7eb0c5364278a1eae0ac376db5c73dda8e476857b3519ad6c0c121904bd WatchSource:0}: Error finding container ad31c7eb0c5364278a1eae0ac376db5c73dda8e476857b3519ad6c0c121904bd: Status 404 returned error can't find the container with id ad31c7eb0c5364278a1eae0ac376db5c73dda8e476857b3519ad6c0c121904bd Oct 30 19:16:29.682223 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:29.682145 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:30.392397 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:30.392364 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qpqdl" event={"ID":"4d9fba76-deca-4eac-908e-03103b2f779e","Type":"ContainerStarted","Data":"ad31c7eb0c5364278a1eae0ac376db5c73dda8e476857b3519ad6c0c121904bd"} Oct 30 19:16:31.210363 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:31.210325 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:31.402607 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:31.402572 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" event={"ID":"c142a503-11c1-4f87-aebb-f68610a7d227","Type":"ContainerStarted","Data":"bc56bf59e4d9e19e978abc4b7be317759811921338109e64cb7cba9851573888"} Oct 30 19:16:31.403005 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:31.402667 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" event={"ID":"c142a503-11c1-4f87-aebb-f68610a7d227","Type":"ContainerStarted","Data":"a7d260ee36a54123a4440613ff5229b91a191e07ef64f310de8cc9474ff53ec5"} Oct 30 19:16:31.403005 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:31.402695 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" event={"ID":"c142a503-11c1-4f87-aebb-f68610a7d227","Type":"ContainerStarted","Data":"029644e488eca042b123606e2399c19d932508e0478a644cf1c0a0a0bf7667b1"} Oct 30 19:16:31.404112 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:31.404082 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qpqdl" event={"ID":"4d9fba76-deca-4eac-908e-03103b2f779e","Type":"ContainerStarted","Data":"a4bca9ea076365424fd971a77bc7a84d5750698ee3cf4bf19b7019aaeaa939da"} Oct 30 19:16:31.404275 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:31.404253 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qpqdl" Oct 30 19:16:31.408927 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:31.408907 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qpqdl" Oct 30 19:16:31.429529 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:31.429506 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:31.517229 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:31.517110 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qpqdl" podStartSLOduration=1.673262958 podStartE2EDuration="3.517097393s" totalImagesPullingTime="1.843834435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:31.516247288 +0000 UTC m=+38.451034814" watchObservedRunningTime="2026-10-30 19:16:31.517097393 +0000 UTC m=+38.451884915" Oct 30 19:16:32.410380 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:32.410345 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" event={"ID":"c142a503-11c1-4f87-aebb-f68610a7d227","Type":"ContainerStarted","Data":"2548a93c0200ddd53afbd42c4af7e9069d967457d57617a6718f72dee8c91aaf"} Oct 30 19:16:32.410879 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:32.410412 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" event={"ID":"c142a503-11c1-4f87-aebb-f68610a7d227","Type":"ContainerStarted","Data":"0624f70ed5564440374e5fa9c1991ac487150e7195790e14ff0d7cce5ff386d0"} Oct 30 19:16:32.410879 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:32.410441 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" event={"ID":"c142a503-11c1-4f87-aebb-f68610a7d227","Type":"ContainerStarted","Data":"6713ae52842e3b1ee1f124a8f2662687e8451091dc4e9fec89c0a0b061657e43"} Oct 30 19:16:32.627756 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:32.627698 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" podStartSLOduration=2.527199623 podStartE2EDuration="6.627681485s" totalImagesPullingTime="4.100481862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:32.578215573 +0000 UTC m=+39.513003098" watchObservedRunningTime="2026-10-30 19:16:32.627681485 +0000 UTC m=+39.562469010" Oct 30 19:16:32.880036 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:32.880003 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:33.124225 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:33.124193 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:33.331880 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:33.331855 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f57c7f55b-mpdsx" Oct 30 19:16:33.393725 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:33.393696 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:33.413357 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:33.413332 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:33.770507 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:33.770418 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:34.060965 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:34.060881 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:34.426092 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:34.426059 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:34.711703 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:34.711600 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:37.541163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.541049 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d7d96bd5-z89jl"] Oct 30 19:16:37.543359 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.543335 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.580369 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.580335 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-serving-cert\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.580518 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.580381 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-config\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.580518 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.580431 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-tmp\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.580592 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.580555 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-oauth-serving-cert\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.580657 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.580625 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgc85\" (UniqueName: \"kubernetes.io/projected/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-kube-api-access-dgc85\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.580695 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.580656 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-oauth-config\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.580695 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.580684 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-service-ca\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.645573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.645525 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 19:16:37.645573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.645544 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 19:16:37.645573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.645566 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 19:16:37.645820 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.645625 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 19:16:37.645820 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.645628 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 19:16:37.648597 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.648574 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d7d96bd5-z89jl"] Oct 30 19:16:37.649937 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.649919 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 19:16:37.650043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.650008 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 19:16:37.681745 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.681707 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-service-ca\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.681745 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.681749 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-serving-cert\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.682067 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.681777 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-config\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.682067 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.681803 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-tmp\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.682067 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.681833 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-oauth-serving-cert\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.682067 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.681858 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgc85\" (UniqueName: \"kubernetes.io/projected/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-kube-api-access-dgc85\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.682067 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.681874 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-oauth-config\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.682310 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.682246 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-tmp\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.682605 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.682583 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-service-ca\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.682605 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.682596 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-oauth-serving-cert\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.682732 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.682663 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-config\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.684338 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.684319 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-oauth-config\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.684405 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.684375 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-serving-cert\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.695115 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.695090 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2795w\"" Oct 30 19:16:37.738857 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.738816 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgc85\" (UniqueName: \"kubernetes.io/projected/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-kube-api-access-dgc85\") pod \"console-5d7d96bd5-z89jl\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.853818 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.853783 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:37.988591 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:37.988555 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:38.226726 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:38.226650 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d7d96bd5-z89jl"] Oct 30 19:16:38.263761 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:16:38.263732 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e46d0a_2d77_4770_b02c_d38ef1a2a7be.slice/crio-249d03708e7c3cbe490d576d57f466979aba436976cc12ed5a112643accd3eae WatchSource:0}: Error finding container 249d03708e7c3cbe490d576d57f466979aba436976cc12ed5a112643accd3eae: Status 404 returned error can't find the container with id 249d03708e7c3cbe490d576d57f466979aba436976cc12ed5a112643accd3eae Oct 30 19:16:38.368307 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:38.368273 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:38.431285 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:38.431249 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7d96bd5-z89jl" event={"ID":"29e46d0a-2d77-4770-b02c-d38ef1a2a7be","Type":"ContainerStarted","Data":"249d03708e7c3cbe490d576d57f466979aba436976cc12ed5a112643accd3eae"} Oct 30 19:16:39.424694 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:39.424654 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f48b5bf56-wq96p" Oct 30 19:16:40.071701 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:40.071659 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:41.304407 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:41.304374 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:41.543898 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:41.543863 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:42.447540 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:42.447498 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7d96bd5-z89jl" event={"ID":"29e46d0a-2d77-4770-b02c-d38ef1a2a7be","Type":"ContainerStarted","Data":"10377ee61ef4c7c91ffc3c0f3cd810ed517cacdf2f6ee04214e25de192e860b2"} Oct 30 19:16:42.569260 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:42.569206 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d7d96bd5-z89jl" podStartSLOduration=2.326611342 podStartE2EDuration="5.569190869s" totalImagesPullingTime="3.242579527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:42.568521096 +0000 UTC m=+49.503308620" watchObservedRunningTime="2026-10-30 19:16:42.569190869 +0000 UTC m=+49.503978393" Oct 30 19:16:42.956874 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:42.956834 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:43.236161 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:43.236077 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:45.233445 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:45.233392 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:45.447658 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:45.447621 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:46.862075 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:46.862041 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:47.125597 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:47.125519 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:47.854104 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:47.854056 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:47.854355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:47.854336 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:47.859176 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:47.859154 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:47.869959 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:47.869930 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-78bdff664b-vnbvk"] Oct 30 19:16:47.872329 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:47.872312 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:47.963917 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:47.963881 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-oauth-serving-cert\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:47.964117 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:47.963932 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-serving-cert\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:47.964187 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:47.964166 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp92m\" (UniqueName: \"kubernetes.io/projected/d7162456-23e1-4cad-88b8-d79e1c0c60ac-kube-api-access-hp92m\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:47.964261 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:47.964246 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-config\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:47.964323 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:47.964267 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7162456-23e1-4cad-88b8-d79e1c0c60ac-tmp\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:47.964378 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:47.964338 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-service-ca\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:47.964430 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:47.964375 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-trusted-ca-bundle\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:47.964479 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:47.964449 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-oauth-config\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.065480 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.065429 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-oauth-config\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.065480 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.065484 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-oauth-serving-cert\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.065733 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.065502 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-serving-cert\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.065733 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.065523 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp92m\" (UniqueName: \"kubernetes.io/projected/d7162456-23e1-4cad-88b8-d79e1c0c60ac-kube-api-access-hp92m\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.065733 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.065554 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-config\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.065733 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.065569 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7162456-23e1-4cad-88b8-d79e1c0c60ac-tmp\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.065733 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.065588 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-service-ca\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.065733 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.065601 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-trusted-ca-bundle\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.066132 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.066101 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7162456-23e1-4cad-88b8-d79e1c0c60ac-tmp\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.066350 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.066327 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-oauth-serving-cert\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.066430 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.066408 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-config\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.066489 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.066447 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-service-ca\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.068087 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.068061 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-serving-cert\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.068210 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.068131 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-oauth-config\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.076286 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.076258 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 19:16:48.077169 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.077148 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78bdff664b-vnbvk"] Oct 30 19:16:48.087274 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.087240 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-trusted-ca-bundle\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.122242 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.122172 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp92m\" (UniqueName: \"kubernetes.io/projected/d7162456-23e1-4cad-88b8-d79e1c0c60ac-kube-api-access-hp92m\") pod \"console-78bdff664b-vnbvk\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.181907 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.181867 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:48.471229 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.471140 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:16:48.553680 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:48.553648 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78bdff664b-vnbvk"] Oct 30 19:16:48.556660 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:16:48.556629 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7162456_23e1_4cad_88b8_d79e1c0c60ac.slice/crio-b9793019b90e9002f380ac70a78edc0133eb8a8ec709d44088f79826d3171d00 WatchSource:0}: Error finding container b9793019b90e9002f380ac70a78edc0133eb8a8ec709d44088f79826d3171d00: Status 404 returned error can't find the container with id b9793019b90e9002f380ac70a78edc0133eb8a8ec709d44088f79826d3171d00 Oct 30 19:16:49.294277 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:49.294244 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:49.471521 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:49.471484 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bdff664b-vnbvk" event={"ID":"d7162456-23e1-4cad-88b8-d79e1c0c60ac","Type":"ContainerStarted","Data":"a8f9f2debcf0e93e89fffcfb51dba4a1430ed734755e28b38fbcaed9ab84349b"} Oct 30 19:16:49.471708 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:49.471541 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bdff664b-vnbvk" event={"ID":"d7162456-23e1-4cad-88b8-d79e1c0c60ac","Type":"ContainerStarted","Data":"b9793019b90e9002f380ac70a78edc0133eb8a8ec709d44088f79826d3171d00"} Oct 30 19:16:49.523292 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:49.523248 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:49.638740 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:49.638679 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-78bdff664b-vnbvk" podStartSLOduration=2.638664103 podStartE2EDuration="2.638664103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:49.588745609 +0000 UTC m=+56.523533134" watchObservedRunningTime="2026-10-30 19:16:49.638664103 +0000 UTC m=+56.573451627" Oct 30 19:16:49.758042 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:49.758008 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:53.365573 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:16:53.365533 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:16:53.811470 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:53.811380 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:54.034277 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:54.034243 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:58.182592 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:58.182548 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:58.183080 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:58.182623 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:58.187339 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:58.187313 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:58.489069 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:58.488970 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:58.510370 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:58.510340 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:16:58.677966 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:58.677930 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d7d96bd5-z89jl"] Oct 30 19:16:58.736772 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:16:58.736742 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:02.025652 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:02.025617 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:02.261656 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:02.261622 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:04.349845 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:04.349797 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:04.592384 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:04.592351 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:19.302677 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:19.302645 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:19.587826 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:19.587789 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:20.051521 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.051424 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hjklm-r9jn7"] Oct 30 19:17:20.051717 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.051686 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-hjklm-r9jn7" podUID="44f7fda2-c7d3-470c-9080-1f931fac9baa" containerName="insights-gathering" containerID="cri-o://8f5d4eeb4c8d73f1db1d35976741926823d65ca1b9ac898a4179fe59cf9ac202" gracePeriod=30 Oct 30 19:17:20.300880 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.300856 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:17:20.329197 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.329161 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-service-ca-bundle\") pod \"44f7fda2-c7d3-470c-9080-1f931fac9baa\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " Oct 30 19:17:20.329594 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.329216 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f7fda2-c7d3-470c-9080-1f931fac9baa-kube-api-access-jh4p5\" (UniqueName: \"kubernetes.io/projected/44f7fda2-c7d3-470c-9080-1f931fac9baa-kube-api-access-jh4p5\") pod \"44f7fda2-c7d3-470c-9080-1f931fac9baa\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " Oct 30 19:17:20.329594 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.329242 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-trusted-ca-bundle\") pod \"44f7fda2-c7d3-470c-9080-1f931fac9baa\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " Oct 30 19:17:20.329594 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.329283 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f7fda2-c7d3-470c-9080-1f931fac9baa-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/44f7fda2-c7d3-470c-9080-1f931fac9baa-archives-path\") pod \"44f7fda2-c7d3-470c-9080-1f931fac9baa\" (UID: \"44f7fda2-c7d3-470c-9080-1f931fac9baa\") " Oct 30 19:17:20.329711 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.329630 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-service-ca-bundle" pod "44f7fda2-c7d3-470c-9080-1f931fac9baa" (UID: "44f7fda2-c7d3-470c-9080-1f931fac9baa"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:20.329711 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.329651 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f7fda2-c7d3-470c-9080-1f931fac9baa-archives-path" pod "44f7fda2-c7d3-470c-9080-1f931fac9baa" (UID: "44f7fda2-c7d3-470c-9080-1f931fac9baa"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:20.329711 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.329640 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-trusted-ca-bundle" pod "44f7fda2-c7d3-470c-9080-1f931fac9baa" (UID: "44f7fda2-c7d3-470c-9080-1f931fac9baa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:20.331553 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.331535 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f7fda2-c7d3-470c-9080-1f931fac9baa-kube-api-access-jh4p5" pod "44f7fda2-c7d3-470c-9080-1f931fac9baa" (UID: "44f7fda2-c7d3-470c-9080-1f931fac9baa"). InnerVolumeSpecName "kube-api-access-jh4p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:20.430719 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.430686 4433 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/44f7fda2-c7d3-470c-9080-1f931fac9baa-archives-path\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:20.430719 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.430712 4433 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-service-ca-bundle\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:20.430719 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.430724 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh4p5\" (UniqueName: \"kubernetes.io/projected/44f7fda2-c7d3-470c-9080-1f931fac9baa-kube-api-access-jh4p5\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:20.430937 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.430734 4433 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f7fda2-c7d3-470c-9080-1f931fac9baa-trusted-ca-bundle\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:20.580973 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.580893 4433 generic.go:356] "Generic (PLEG): container finished" podID="44f7fda2-c7d3-470c-9080-1f931fac9baa" containerID="8f5d4eeb4c8d73f1db1d35976741926823d65ca1b9ac898a4179fe59cf9ac202" exitCode=2 Oct 30 19:17:20.580973 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.580936 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hjklm-r9jn7" Oct 30 19:17:20.580973 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.580955 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hjklm-r9jn7" event={"ID":"44f7fda2-c7d3-470c-9080-1f931fac9baa","Type":"ContainerDied","Data":"8f5d4eeb4c8d73f1db1d35976741926823d65ca1b9ac898a4179fe59cf9ac202"} Oct 30 19:17:20.581202 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.581007 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hjklm-r9jn7" event={"ID":"44f7fda2-c7d3-470c-9080-1f931fac9baa","Type":"ContainerDied","Data":"e6d43012785906e18c9802e1072e7f84a90b3dd1c4ff63b59cb094da8141f808"} Oct 30 19:17:20.581202 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.581037 4433 scope.go:122] "RemoveContainer" containerID="8f5d4eeb4c8d73f1db1d35976741926823d65ca1b9ac898a4179fe59cf9ac202" Oct 30 19:17:20.589386 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.589365 4433 scope.go:122] "RemoveContainer" containerID="8f5d4eeb4c8d73f1db1d35976741926823d65ca1b9ac898a4179fe59cf9ac202" Oct 30 19:17:20.589643 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:17:20.589623 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5d4eeb4c8d73f1db1d35976741926823d65ca1b9ac898a4179fe59cf9ac202\": container with ID starting with 8f5d4eeb4c8d73f1db1d35976741926823d65ca1b9ac898a4179fe59cf9ac202 not found: ID does not exist" containerID="8f5d4eeb4c8d73f1db1d35976741926823d65ca1b9ac898a4179fe59cf9ac202" Oct 30 19:17:20.589718 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.589657 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5d4eeb4c8d73f1db1d35976741926823d65ca1b9ac898a4179fe59cf9ac202"} err="failed to get container status \"8f5d4eeb4c8d73f1db1d35976741926823d65ca1b9ac898a4179fe59cf9ac202\": rpc error: code = NotFound desc = could not find container \"8f5d4eeb4c8d73f1db1d35976741926823d65ca1b9ac898a4179fe59cf9ac202\": container with ID starting with 8f5d4eeb4c8d73f1db1d35976741926823d65ca1b9ac898a4179fe59cf9ac202 not found: ID does not exist" Oct 30 19:17:20.742544 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.742506 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hjklm-r9jn7"] Oct 30 19:17:20.746207 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:20.746182 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-hjklm-r9jn7"] Oct 30 19:17:21.211340 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:21.211303 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f7fda2-c7d3-470c-9080-1f931fac9baa" path="/var/lib/kubelet/pods/44f7fda2-c7d3-470c-9080-1f931fac9baa/volumes" Oct 30 19:17:23.365419 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:17:23.365379 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:17:23.697827 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:23.697723 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5d7d96bd5-z89jl" podUID="29e46d0a-2d77-4770-b02c-d38ef1a2a7be" containerName="console" containerID="cri-o://10377ee61ef4c7c91ffc3c0f3cd810ed517cacdf2f6ee04214e25de192e860b2" gracePeriod=15 Oct 30 19:17:23.940992 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:23.940957 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:17:23.956555 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:23.956491 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-serving-cert\") pod \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " Oct 30 19:17:23.956555 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:23.956537 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-kube-api-access-dgc85\" (UniqueName: \"kubernetes.io/projected/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-kube-api-access-dgc85\") pod \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " Oct 30 19:17:23.956728 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:23.956581 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-service-ca\") pod \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " Oct 30 19:17:23.956728 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:23.956602 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-oauth-config\") pod \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " Oct 30 19:17:23.956728 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:23.956637 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-tmp\" (UniqueName: \"kubernetes.io/empty-dir/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-tmp\") pod \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " Oct 30 19:17:23.956728 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:23.956676 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-oauth-serving-cert\") pod \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " Oct 30 19:17:23.956905 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:23.956731 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-config\" (UniqueName: \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-config\") pod \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\" (UID: \"29e46d0a-2d77-4770-b02c-d38ef1a2a7be\") " Oct 30 19:17:23.957165 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:23.957120 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-tmp" pod "29e46d0a-2d77-4770-b02c-d38ef1a2a7be" (UID: "29e46d0a-2d77-4770-b02c-d38ef1a2a7be"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:23.957324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:23.957264 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-service-ca" pod "29e46d0a-2d77-4770-b02c-d38ef1a2a7be" (UID: "29e46d0a-2d77-4770-b02c-d38ef1a2a7be"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:23.957379 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:23.957341 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-config" pod "29e46d0a-2d77-4770-b02c-d38ef1a2a7be" (UID: "29e46d0a-2d77-4770-b02c-d38ef1a2a7be"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:23.957379 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:23.957347 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-oauth-serving-cert" pod "29e46d0a-2d77-4770-b02c-d38ef1a2a7be" (UID: "29e46d0a-2d77-4770-b02c-d38ef1a2a7be"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:23.958896 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:23.958876 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-serving-cert" pod "29e46d0a-2d77-4770-b02c-d38ef1a2a7be" (UID: "29e46d0a-2d77-4770-b02c-d38ef1a2a7be"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:23.959124 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:23.959100 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-oauth-config" pod "29e46d0a-2d77-4770-b02c-d38ef1a2a7be" (UID: "29e46d0a-2d77-4770-b02c-d38ef1a2a7be"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:23.959213 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:23.959144 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-kube-api-access-dgc85" pod "29e46d0a-2d77-4770-b02c-d38ef1a2a7be" (UID: "29e46d0a-2d77-4770-b02c-d38ef1a2a7be"). InnerVolumeSpecName "kube-api-access-dgc85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:24.057845 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:24.057813 4433 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-config\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:24.057845 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:24.057840 4433 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-serving-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:24.057845 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:24.057850 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgc85\" (UniqueName: \"kubernetes.io/projected/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-kube-api-access-dgc85\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:24.058144 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:24.057861 4433 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:24.058144 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:24.057870 4433 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-console-oauth-config\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:24.058144 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:24.057881 4433 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-tmp\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:24.058144 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:24.057893 4433 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29e46d0a-2d77-4770-b02c-d38ef1a2a7be-oauth-serving-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:24.595701 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:24.595667 4433 generic.go:356] "Generic (PLEG): container finished" podID="29e46d0a-2d77-4770-b02c-d38ef1a2a7be" containerID="10377ee61ef4c7c91ffc3c0f3cd810ed517cacdf2f6ee04214e25de192e860b2" exitCode=0 Oct 30 19:17:24.596212 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:24.595712 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7d96bd5-z89jl" Oct 30 19:17:24.596212 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:24.595729 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7d96bd5-z89jl" event={"ID":"29e46d0a-2d77-4770-b02c-d38ef1a2a7be","Type":"ContainerDied","Data":"10377ee61ef4c7c91ffc3c0f3cd810ed517cacdf2f6ee04214e25de192e860b2"} Oct 30 19:17:24.596212 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:24.595779 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7d96bd5-z89jl" event={"ID":"29e46d0a-2d77-4770-b02c-d38ef1a2a7be","Type":"ContainerDied","Data":"249d03708e7c3cbe490d576d57f466979aba436976cc12ed5a112643accd3eae"} Oct 30 19:17:24.596212 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:24.595805 4433 scope.go:122] "RemoveContainer" containerID="10377ee61ef4c7c91ffc3c0f3cd810ed517cacdf2f6ee04214e25de192e860b2" Oct 30 19:17:24.604168 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:24.604150 4433 scope.go:122] "RemoveContainer" containerID="10377ee61ef4c7c91ffc3c0f3cd810ed517cacdf2f6ee04214e25de192e860b2" Oct 30 19:17:24.604461 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:17:24.604442 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10377ee61ef4c7c91ffc3c0f3cd810ed517cacdf2f6ee04214e25de192e860b2\": container with ID starting with 10377ee61ef4c7c91ffc3c0f3cd810ed517cacdf2f6ee04214e25de192e860b2 not found: ID does not exist" containerID="10377ee61ef4c7c91ffc3c0f3cd810ed517cacdf2f6ee04214e25de192e860b2" Oct 30 19:17:24.604509 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:24.604471 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10377ee61ef4c7c91ffc3c0f3cd810ed517cacdf2f6ee04214e25de192e860b2"} err="failed to get container status \"10377ee61ef4c7c91ffc3c0f3cd810ed517cacdf2f6ee04214e25de192e860b2\": rpc error: code = NotFound desc = could not find container \"10377ee61ef4c7c91ffc3c0f3cd810ed517cacdf2f6ee04214e25de192e860b2\": container with ID starting with 10377ee61ef4c7c91ffc3c0f3cd810ed517cacdf2f6ee04214e25de192e860b2 not found: ID does not exist" Oct 30 19:17:24.758145 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:24.758110 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d7d96bd5-z89jl"] Oct 30 19:17:24.807479 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:24.807439 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d7d96bd5-z89jl"] Oct 30 19:17:25.210953 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:25.210919 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e46d0a-2d77-4770-b02c-d38ef1a2a7be" path="/var/lib/kubelet/pods/29e46d0a-2d77-4770-b02c-d38ef1a2a7be/volumes" Oct 30 19:17:43.007224 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:43.007168 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:43.223951 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:43.223891 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:43.465247 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:43.465216 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:50.429758 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:50.429719 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:53.256573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:53.256530 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:53.366087 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:17:53.366051 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:17:55.582013 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:55.581930 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:55.807350 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:17:55.807320 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:04.313318 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:04.313281 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:04.551420 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:04.551384 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:06.335693 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.335660 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bd8fbf98f-pjb9r"] Oct 30 19:18:06.336121 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.335913 4433 cpu_manager.go:414] "containerMap: removing container" podUID="29e46d0a-2d77-4770-b02c-d38ef1a2a7be" containerName="console" Oct 30 19:18:06.336121 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.335923 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e46d0a-2d77-4770-b02c-d38ef1a2a7be" containerName="console" Oct 30 19:18:06.336121 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.335936 4433 cpu_manager.go:414] "containerMap: removing container" podUID="44f7fda2-c7d3-470c-9080-1f931fac9baa" containerName="insights-gathering" Oct 30 19:18:06.336121 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.335941 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f7fda2-c7d3-470c-9080-1f931fac9baa" containerName="insights-gathering" Oct 30 19:18:06.336121 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.336027 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e46d0a-2d77-4770-b02c-d38ef1a2a7be" containerName="console" Oct 30 19:18:06.336121 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.336038 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f7fda2-c7d3-470c-9080-1f931fac9baa" containerName="insights-gathering" Oct 30 19:18:06.338894 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.338878 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.441374 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.441336 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bd8fbf98f-pjb9r"] Oct 30 19:18:06.470073 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.470033 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04c30a05-d824-4b3b-a05a-78760bf57fc9-tmp\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.470073 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.470070 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04c30a05-d824-4b3b-a05a-78760bf57fc9-service-ca\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.470292 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.470147 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04c30a05-d824-4b3b-a05a-78760bf57fc9-console-serving-cert\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.470292 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.470163 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04c30a05-d824-4b3b-a05a-78760bf57fc9-console-config\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.470292 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.470180 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04c30a05-d824-4b3b-a05a-78760bf57fc9-oauth-serving-cert\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.470399 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.470298 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04c30a05-d824-4b3b-a05a-78760bf57fc9-console-oauth-config\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.470399 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.470329 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wscnf\" (UniqueName: \"kubernetes.io/projected/04c30a05-d824-4b3b-a05a-78760bf57fc9-kube-api-access-wscnf\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.470399 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.470354 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04c30a05-d824-4b3b-a05a-78760bf57fc9-trusted-ca-bundle\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.571436 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.571344 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04c30a05-d824-4b3b-a05a-78760bf57fc9-console-serving-cert\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.571436 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.571441 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04c30a05-d824-4b3b-a05a-78760bf57fc9-console-config\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.571679 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.571472 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04c30a05-d824-4b3b-a05a-78760bf57fc9-oauth-serving-cert\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.571679 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.571516 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04c30a05-d824-4b3b-a05a-78760bf57fc9-console-oauth-config\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.571679 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.571540 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wscnf\" (UniqueName: \"kubernetes.io/projected/04c30a05-d824-4b3b-a05a-78760bf57fc9-kube-api-access-wscnf\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.571679 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.571571 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04c30a05-d824-4b3b-a05a-78760bf57fc9-trusted-ca-bundle\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.571910 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.571871 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04c30a05-d824-4b3b-a05a-78760bf57fc9-tmp\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.572017 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.571922 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04c30a05-d824-4b3b-a05a-78760bf57fc9-service-ca\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.572331 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.572275 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04c30a05-d824-4b3b-a05a-78760bf57fc9-tmp\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.572441 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.572331 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04c30a05-d824-4b3b-a05a-78760bf57fc9-console-config\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.572441 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.572337 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04c30a05-d824-4b3b-a05a-78760bf57fc9-oauth-serving-cert\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.572573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.572470 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04c30a05-d824-4b3b-a05a-78760bf57fc9-trusted-ca-bundle\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.572655 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.572631 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04c30a05-d824-4b3b-a05a-78760bf57fc9-service-ca\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.574096 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.574070 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04c30a05-d824-4b3b-a05a-78760bf57fc9-console-serving-cert\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.574096 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.574084 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04c30a05-d824-4b3b-a05a-78760bf57fc9-console-oauth-config\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.629729 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.629658 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wscnf\" (UniqueName: \"kubernetes.io/projected/04c30a05-d824-4b3b-a05a-78760bf57fc9-kube-api-access-wscnf\") pod \"console-5bd8fbf98f-pjb9r\" (UID: \"04c30a05-d824-4b3b-a05a-78760bf57fc9\") " pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.649734 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.649699 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:06.694768 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.694728 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:06.955365 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:06.955221 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:07.021538 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:07.021503 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bd8fbf98f-pjb9r"] Oct 30 19:18:07.025314 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:18:07.025288 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c30a05_d824_4b3b_a05a_78760bf57fc9.slice/crio-3d636a428e7d6a52f82c2a4ff2d376cdcc63f30491b42fa006e1d5dc9b62b073 WatchSource:0}: Error finding container 3d636a428e7d6a52f82c2a4ff2d376cdcc63f30491b42fa006e1d5dc9b62b073: Status 404 returned error can't find the container with id 3d636a428e7d6a52f82c2a4ff2d376cdcc63f30491b42fa006e1d5dc9b62b073 Oct 30 19:18:07.211156 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:07.211070 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:07.743462 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:07.743420 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bd8fbf98f-pjb9r" event={"ID":"04c30a05-d824-4b3b-a05a-78760bf57fc9","Type":"ContainerStarted","Data":"caa82b18324e88541ca2543fa9499a0c4faf4165e800963de8c618a1a0e1da72"} Oct 30 19:18:07.743951 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:07.743471 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bd8fbf98f-pjb9r" event={"ID":"04c30a05-d824-4b3b-a05a-78760bf57fc9","Type":"ContainerStarted","Data":"3d636a428e7d6a52f82c2a4ff2d376cdcc63f30491b42fa006e1d5dc9b62b073"} Oct 30 19:18:07.860309 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:07.860257 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5bd8fbf98f-pjb9r" podStartSLOduration=1.860241198 podStartE2EDuration="1.860241198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:07.859672141 +0000 UTC m=+134.794459666" watchObservedRunningTime="2026-10-30 19:18:07.860241198 +0000 UTC m=+134.795028723" Oct 30 19:18:16.650166 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:16.650111 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:16.650779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:16.650193 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:16.655133 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:16.655108 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:16.776043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:16.776015 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bd8fbf98f-pjb9r" Oct 30 19:18:16.919280 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:16.919200 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78bdff664b-vnbvk"] Oct 30 19:18:23.365783 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:18:23.365739 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:18:37.116525 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:37.116479 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:41.943926 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:41.943857 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-78bdff664b-vnbvk" podUID="d7162456-23e1-4cad-88b8-d79e1c0c60ac" containerName="console" containerID="cri-o://a8f9f2debcf0e93e89fffcfb51dba4a1430ed734755e28b38fbcaed9ab84349b" gracePeriod=15 Oct 30 19:18:42.201488 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.201430 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:18:42.234509 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.234478 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-oauth-config\") pod \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " Oct 30 19:18:42.234509 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.234512 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-config\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-config\") pod \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " Oct 30 19:18:42.234705 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.234539 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-oauth-serving-cert\") pod \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " Oct 30 19:18:42.234705 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.234578 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-service-ca\") pod \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " Oct 30 19:18:42.234705 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.234607 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-serving-cert\") pod \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " Oct 30 19:18:42.234705 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.234637 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7162456-23e1-4cad-88b8-d79e1c0c60ac-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7162456-23e1-4cad-88b8-d79e1c0c60ac-tmp\") pod \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " Oct 30 19:18:42.234991 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.234952 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7162456-23e1-4cad-88b8-d79e1c0c60ac-tmp" pod "d7162456-23e1-4cad-88b8-d79e1c0c60ac" (UID: "d7162456-23e1-4cad-88b8-d79e1c0c60ac"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:42.235111 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.235020 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-config" pod "d7162456-23e1-4cad-88b8-d79e1c0c60ac" (UID: "d7162456-23e1-4cad-88b8-d79e1c0c60ac"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:42.235168 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.235062 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7162456-23e1-4cad-88b8-d79e1c0c60ac-kube-api-access-hp92m\" (UniqueName: \"kubernetes.io/projected/d7162456-23e1-4cad-88b8-d79e1c0c60ac-kube-api-access-hp92m\") pod \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " Oct 30 19:18:42.235168 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.235157 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-trusted-ca-bundle\") pod \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\" (UID: \"d7162456-23e1-4cad-88b8-d79e1c0c60ac\") " Oct 30 19:18:42.235241 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.235061 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-service-ca" pod "d7162456-23e1-4cad-88b8-d79e1c0c60ac" (UID: "d7162456-23e1-4cad-88b8-d79e1c0c60ac"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:42.235241 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.235076 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-oauth-serving-cert" pod "d7162456-23e1-4cad-88b8-d79e1c0c60ac" (UID: "d7162456-23e1-4cad-88b8-d79e1c0c60ac"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:42.235548 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.235528 4433 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7162456-23e1-4cad-88b8-d79e1c0c60ac-tmp\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:42.235629 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.235555 4433 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-config\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:42.235629 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.235560 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-trusted-ca-bundle" pod "d7162456-23e1-4cad-88b8-d79e1c0c60ac" (UID: "d7162456-23e1-4cad-88b8-d79e1c0c60ac"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:42.235629 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.235570 4433 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-oauth-serving-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:42.235629 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.235586 4433 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:42.236975 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.236950 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-oauth-config" pod "d7162456-23e1-4cad-88b8-d79e1c0c60ac" (UID: "d7162456-23e1-4cad-88b8-d79e1c0c60ac"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:42.237082 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.237008 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-serving-cert" pod "d7162456-23e1-4cad-88b8-d79e1c0c60ac" (UID: "d7162456-23e1-4cad-88b8-d79e1c0c60ac"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:42.237252 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.237233 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7162456-23e1-4cad-88b8-d79e1c0c60ac-kube-api-access-hp92m" pod "d7162456-23e1-4cad-88b8-d79e1c0c60ac" (UID: "d7162456-23e1-4cad-88b8-d79e1c0c60ac"). InnerVolumeSpecName "kube-api-access-hp92m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:42.336465 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.336433 4433 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-oauth-config\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:42.336465 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.336462 4433 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7162456-23e1-4cad-88b8-d79e1c0c60ac-console-serving-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:42.336676 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.336471 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp92m\" (UniqueName: \"kubernetes.io/projected/d7162456-23e1-4cad-88b8-d79e1c0c60ac-kube-api-access-hp92m\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:42.336676 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.336493 4433 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7162456-23e1-4cad-88b8-d79e1c0c60ac-trusted-ca-bundle\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:42.859741 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.859705 4433 generic.go:356] "Generic (PLEG): container finished" podID="d7162456-23e1-4cad-88b8-d79e1c0c60ac" containerID="a8f9f2debcf0e93e89fffcfb51dba4a1430ed734755e28b38fbcaed9ab84349b" exitCode=0 Oct 30 19:18:42.859920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.859763 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bdff664b-vnbvk" Oct 30 19:18:42.859920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.859806 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bdff664b-vnbvk" event={"ID":"d7162456-23e1-4cad-88b8-d79e1c0c60ac","Type":"ContainerDied","Data":"a8f9f2debcf0e93e89fffcfb51dba4a1430ed734755e28b38fbcaed9ab84349b"} Oct 30 19:18:42.859920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.859867 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bdff664b-vnbvk" event={"ID":"d7162456-23e1-4cad-88b8-d79e1c0c60ac","Type":"ContainerDied","Data":"b9793019b90e9002f380ac70a78edc0133eb8a8ec709d44088f79826d3171d00"} Oct 30 19:18:42.859920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.859887 4433 scope.go:122] "RemoveContainer" containerID="a8f9f2debcf0e93e89fffcfb51dba4a1430ed734755e28b38fbcaed9ab84349b" Oct 30 19:18:42.868407 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.868385 4433 scope.go:122] "RemoveContainer" containerID="a8f9f2debcf0e93e89fffcfb51dba4a1430ed734755e28b38fbcaed9ab84349b" Oct 30 19:18:42.868671 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:18:42.868652 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f9f2debcf0e93e89fffcfb51dba4a1430ed734755e28b38fbcaed9ab84349b\": container with ID starting with a8f9f2debcf0e93e89fffcfb51dba4a1430ed734755e28b38fbcaed9ab84349b not found: ID does not exist" containerID="a8f9f2debcf0e93e89fffcfb51dba4a1430ed734755e28b38fbcaed9ab84349b" Oct 30 19:18:42.868739 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:42.868681 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f9f2debcf0e93e89fffcfb51dba4a1430ed734755e28b38fbcaed9ab84349b"} err="failed to get container status \"a8f9f2debcf0e93e89fffcfb51dba4a1430ed734755e28b38fbcaed9ab84349b\": rpc error: code = NotFound desc = could not find container \"a8f9f2debcf0e93e89fffcfb51dba4a1430ed734755e28b38fbcaed9ab84349b\": container with ID starting with a8f9f2debcf0e93e89fffcfb51dba4a1430ed734755e28b38fbcaed9ab84349b not found: ID does not exist" Oct 30 19:18:43.023045 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:43.023014 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78bdff664b-vnbvk"] Oct 30 19:18:43.071624 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:43.071595 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78bdff664b-vnbvk"] Oct 30 19:18:43.211187 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:43.211111 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7162456-23e1-4cad-88b8-d79e1c0c60ac" path="/var/lib/kubelet/pods/d7162456-23e1-4cad-88b8-d79e1c0c60ac/volumes" Oct 30 19:18:43.605861 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:43.605829 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:53.366111 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:18:53.366073 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:18:53.595366 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:18:53.595332 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:23.364365 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:19:23.364327 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:19:37.089035 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:19:37.088939 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:37.334327 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:19:37.334291 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:43.939606 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:19:43.939570 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:53.366283 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:19:53.366238 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:19:53.940897 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:19:53.940844 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:03.886011 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:03.885956 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:05.372877 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:05.372838 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:05.779486 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:05.779401 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-sntvz"] Oct 30 19:20:05.779670 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:05.779659 4433 cpu_manager.go:414] "containerMap: removing container" podUID="d7162456-23e1-4cad-88b8-d79e1c0c60ac" containerName="console" Oct 30 19:20:05.779713 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:05.779671 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7162456-23e1-4cad-88b8-d79e1c0c60ac" containerName="console" Oct 30 19:20:05.779745 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:05.779730 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7162456-23e1-4cad-88b8-d79e1c0c60ac" containerName="console" Oct 30 19:20:05.782403 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:05.782388 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-sntvz" Oct 30 19:20:05.829732 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:05.829689 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnjrd\" (UniqueName: \"kubernetes.io/projected/88aae99c-049d-493e-a54d-3cbddd5f316c-kube-api-access-cnjrd\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-sntvz\" (UID: \"88aae99c-049d-493e-a54d-3cbddd5f316c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-sntvz" Oct 30 19:20:05.829897 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:05.829747 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88aae99c-049d-493e-a54d-3cbddd5f316c-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-sntvz\" (UID: \"88aae99c-049d-493e-a54d-3cbddd5f316c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-sntvz" Oct 30 19:20:05.838597 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:05.838575 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:05.884875 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:05.884842 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 19:20:05.885153 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:05.885132 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7m2hx\"" Oct 30 19:20:05.886081 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:05.886064 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-sntvz"] Oct 30 19:20:05.930864 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:05.930828 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnjrd\" (UniqueName: \"kubernetes.io/projected/88aae99c-049d-493e-a54d-3cbddd5f316c-kube-api-access-cnjrd\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-sntvz\" (UID: \"88aae99c-049d-493e-a54d-3cbddd5f316c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-sntvz" Oct 30 19:20:05.930864 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:05.930870 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88aae99c-049d-493e-a54d-3cbddd5f316c-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-sntvz\" (UID: \"88aae99c-049d-493e-a54d-3cbddd5f316c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-sntvz" Oct 30 19:20:05.931236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:05.931218 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88aae99c-049d-493e-a54d-3cbddd5f316c-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-sntvz\" (UID: \"88aae99c-049d-493e-a54d-3cbddd5f316c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-sntvz" Oct 30 19:20:05.987996 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:05.987947 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnjrd\" (UniqueName: \"kubernetes.io/projected/88aae99c-049d-493e-a54d-3cbddd5f316c-kube-api-access-cnjrd\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-sntvz\" (UID: \"88aae99c-049d-493e-a54d-3cbddd5f316c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-sntvz" Oct 30 19:20:06.092865 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:06.092830 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-sntvz" Oct 30 19:20:06.517457 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:06.517366 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-sntvz"] Oct 30 19:20:06.521761 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:20:06.521732 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88aae99c_049d_493e_a54d_3cbddd5f316c.slice/crio-0d87460729d746154f9866d3e2ea0fe0ef376dbd8acd7866e25251688806d54d WatchSource:0}: Error finding container 0d87460729d746154f9866d3e2ea0fe0ef376dbd8acd7866e25251688806d54d: Status 404 returned error can't find the container with id 0d87460729d746154f9866d3e2ea0fe0ef376dbd8acd7866e25251688806d54d Oct 30 19:20:07.099141 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:07.099104 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:07.134114 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:07.134081 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-sntvz" event={"ID":"88aae99c-049d-493e-a54d-3cbddd5f316c","Type":"ContainerStarted","Data":"0d87460729d746154f9866d3e2ea0fe0ef376dbd8acd7866e25251688806d54d"} Oct 30 19:20:07.323612 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:07.323573 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:10.555790 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:10.555751 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:10.783503 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:10.783427 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:11.151087 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:11.151055 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-sntvz" event={"ID":"88aae99c-049d-493e-a54d-3cbddd5f316c","Type":"ContainerStarted","Data":"0f77b2a69b28322e3ab1799d02bf57380b79f69062d85eea24ec3b142d7ce4f2"} Oct 30 19:20:11.367037 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:11.366966 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-sntvz" podStartSLOduration=2.564885467 podStartE2EDuration="6.36695149s" totalImagesPullingTime="3.802066023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:11.317086373 +0000 UTC m=+258.251873893" watchObservedRunningTime="2026-10-30 19:20:11.36695149 +0000 UTC m=+258.301739014" Oct 30 19:20:11.904102 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:11.904067 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:12.197493 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:12.197419 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:12.430726 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:12.430696 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:13.910689 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:13.910653 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:15.570263 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:15.570228 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:22.542029 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:22.541991 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-k7v8f"] Oct 30 19:20:22.545340 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:22.545322 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k7v8f" Oct 30 19:20:22.647274 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:22.647242 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 19:20:22.647274 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:22.647242 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 19:20:22.647466 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:22.647253 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-l686h\"" Oct 30 19:20:22.648297 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:22.648279 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-k7v8f"] Oct 30 19:20:22.658931 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:22.658905 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a0fbd27-970e-47a8-8515-decc9f38477b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-k7v8f\" (UID: \"8a0fbd27-970e-47a8-8515-decc9f38477b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k7v8f" Oct 30 19:20:22.659050 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:22.658941 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtfv4\" (UniqueName: \"kubernetes.io/projected/8a0fbd27-970e-47a8-8515-decc9f38477b-kube-api-access-gtfv4\") pod \"cert-manager-webhook-5dbbbf75cc-k7v8f\" (UID: \"8a0fbd27-970e-47a8-8515-decc9f38477b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k7v8f" Oct 30 19:20:22.760242 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:22.760209 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a0fbd27-970e-47a8-8515-decc9f38477b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-k7v8f\" (UID: \"8a0fbd27-970e-47a8-8515-decc9f38477b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k7v8f" Oct 30 19:20:22.760242 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:22.760245 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtfv4\" (UniqueName: \"kubernetes.io/projected/8a0fbd27-970e-47a8-8515-decc9f38477b-kube-api-access-gtfv4\") pod \"cert-manager-webhook-5dbbbf75cc-k7v8f\" (UID: \"8a0fbd27-970e-47a8-8515-decc9f38477b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k7v8f" Oct 30 19:20:22.817554 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:22.817525 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a0fbd27-970e-47a8-8515-decc9f38477b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-k7v8f\" (UID: \"8a0fbd27-970e-47a8-8515-decc9f38477b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k7v8f" Oct 30 19:20:22.817796 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:22.817776 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtfv4\" (UniqueName: \"kubernetes.io/projected/8a0fbd27-970e-47a8-8515-decc9f38477b-kube-api-access-gtfv4\") pod \"cert-manager-webhook-5dbbbf75cc-k7v8f\" (UID: \"8a0fbd27-970e-47a8-8515-decc9f38477b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k7v8f" Oct 30 19:20:22.870817 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:22.870777 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k7v8f" Oct 30 19:20:23.237656 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:23.237585 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-k7v8f"] Oct 30 19:20:23.240755 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:20:23.240723 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a0fbd27_970e_47a8_8515_decc9f38477b.slice/crio-d1067c172b38d0b6442add30ed41e3070a53a5bc1d74b3677c6b10f7ef0ee82d WatchSource:0}: Error finding container d1067c172b38d0b6442add30ed41e3070a53a5bc1d74b3677c6b10f7ef0ee82d: Status 404 returned error can't find the container with id d1067c172b38d0b6442add30ed41e3070a53a5bc1d74b3677c6b10f7ef0ee82d Oct 30 19:20:23.364580 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:20:23.364543 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:20:23.473074 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:23.473038 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:24.198219 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:24.198186 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k7v8f" event={"ID":"8a0fbd27-970e-47a8-8515-decc9f38477b","Type":"ContainerStarted","Data":"d1067c172b38d0b6442add30ed41e3070a53a5bc1d74b3677c6b10f7ef0ee82d"} Oct 30 19:20:26.209526 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:26.209484 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:26.442398 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:26.442358 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:27.212374 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:27.212283 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k7v8f" event={"ID":"8a0fbd27-970e-47a8-8515-decc9f38477b","Type":"ContainerStarted","Data":"2351d769d5e72677130f8538baca9f82867c099b5537420b8840f59ce029e4c1"} Oct 30 19:20:27.212764 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:27.212429 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k7v8f" Oct 30 19:20:27.326442 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:27.326380 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k7v8f" podStartSLOduration=1.646177193 podStartE2EDuration="5.326362658s" totalImagesPullingTime="3.680185465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:27.325330757 +0000 UTC m=+274.260118281" watchObservedRunningTime="2026-10-30 19:20:27.326362658 +0000 UTC m=+274.261150182" Oct 30 19:20:27.554972 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:27.554890 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:27.800750 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:27.800714 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:33.218224 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:33.218192 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k7v8f" Oct 30 19:20:34.017696 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:34.017658 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:34.250291 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:34.250253 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:36.323107 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:36.323077 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:44.276177 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:44.276140 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:44.499694 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:44.499665 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:48.961355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:48.961317 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:49.218996 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.218907 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:49.728766 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.728734 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:49.846639 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.846600 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg"] Oct 30 19:20:49.852958 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.852940 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:49.956144 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.956105 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 19:20:49.956329 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.956153 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 19:20:49.956329 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.956146 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 19:20:49.956329 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.956141 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2h29g\"" Oct 30 19:20:49.956329 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.956175 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 19:20:49.956329 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.956218 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 19:20:49.980818 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.980650 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg"] Oct 30 19:20:49.982914 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.982877 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wnbq\" (UniqueName: \"kubernetes.io/projected/5c5a1734-680c-472c-a544-723789285116-kube-api-access-8wnbq\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:49.983055 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.982970 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c5a1734-680c-472c-a544-723789285116-tmp\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:49.983055 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.983032 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c5a1734-680c-472c-a544-723789285116-webhook-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:49.983184 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.983057 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5c5a1734-680c-472c-a544-723789285116-manifests\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:49.983184 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.983075 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c5a1734-680c-472c-a544-723789285116-metrics-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:49.983184 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:49.983106 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5c5a1734-680c-472c-a544-723789285116-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:50.073238 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.073156 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:50.084281 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.084255 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wnbq\" (UniqueName: \"kubernetes.io/projected/5c5a1734-680c-472c-a544-723789285116-kube-api-access-8wnbq\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:50.084369 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.084297 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c5a1734-680c-472c-a544-723789285116-tmp\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:50.084426 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.084408 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c5a1734-680c-472c-a544-723789285116-webhook-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:50.084461 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.084444 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5c5a1734-680c-472c-a544-723789285116-manifests\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:50.084502 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.084467 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c5a1734-680c-472c-a544-723789285116-metrics-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:50.084630 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.084611 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5c5a1734-680c-472c-a544-723789285116-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:50.084782 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.084765 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5c5a1734-680c-472c-a544-723789285116-manifests\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:50.086629 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.086602 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c5a1734-680c-472c-a544-723789285116-tmp\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:50.086876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.086859 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5c5a1734-680c-472c-a544-723789285116-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:50.086941 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.086916 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c5a1734-680c-472c-a544-723789285116-webhook-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:50.087025 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.086948 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c5a1734-680c-472c-a544-723789285116-metrics-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:50.153177 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.153140 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wnbq\" (UniqueName: \"kubernetes.io/projected/5c5a1734-680c-472c-a544-723789285116-kube-api-access-8wnbq\") pod \"opendatahub-operator-controller-manager-69bfcc8945-6f4hg\" (UID: \"5c5a1734-680c-472c-a544-723789285116\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:50.167025 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.166995 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:50.394657 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.394621 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:50.659255 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.659051 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:50.659782 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:50.659740 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg"] Oct 30 19:20:50.663779 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:20:50.663754 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5a1734_680c_472c_a544_723789285116.slice/crio-0ffca906e55c8573248c39fcdba19225f36f55293f0db6b43d0a25e5385161fa WatchSource:0}: Error finding container 0ffca906e55c8573248c39fcdba19225f36f55293f0db6b43d0a25e5385161fa: Status 404 returned error can't find the container with id 0ffca906e55c8573248c39fcdba19225f36f55293f0db6b43d0a25e5385161fa Oct 30 19:20:51.295928 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:51.295897 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" event={"ID":"5c5a1734-680c-472c-a544-723789285116","Type":"ContainerStarted","Data":"0ffca906e55c8573248c39fcdba19225f36f55293f0db6b43d0a25e5385161fa"} Oct 30 19:20:53.369104 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:20:53.369062 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:20:54.294069 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:54.294037 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:55.314157 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:55.314123 4433 generic.go:356] "Generic (PLEG): container finished" podID="5c5a1734-680c-472c-a544-723789285116" containerID="af387bd49ec405640dff3fdd5e5822c53bb55c00769d65d8238ad30b1467793f" exitCode=0 Oct 30 19:20:55.314558 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:55.314216 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" event={"ID":"5c5a1734-680c-472c-a544-723789285116","Type":"ContainerDied","Data":"af387bd49ec405640dff3fdd5e5822c53bb55c00769d65d8238ad30b1467793f"} Oct 30 19:20:55.314747 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:55.314731 4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:20:56.320063 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:56.320022 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" event={"ID":"5c5a1734-680c-472c-a544-723789285116","Type":"ContainerStarted","Data":"42e5dd6d121247bc019f1dafb7aef6dd36459a64480cef32c8448c4f97858b10"} Oct 30 19:20:56.320437 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:56.320164 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:20:56.549037 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:20:56.548923 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" podStartSLOduration=2.46561332 podStartE2EDuration="7.548909888s" totalImagesPullingTime="4.628540601s" totalInitContainerRuntime="454.755967ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:56.499101021 +0000 UTC m=+303.433888545" watchObservedRunningTime="2026-10-30 19:20:56.548909888 +0000 UTC m=+303.483697412" Oct 30 19:21:06.357649 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:06.357613 4433 generic.go:356] "Generic (PLEG): container finished" podID="5c5a1734-680c-472c-a544-723789285116" containerID="42e5dd6d121247bc019f1dafb7aef6dd36459a64480cef32c8448c4f97858b10" exitCode=1 Oct 30 19:21:06.358146 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:06.357687 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" event={"ID":"5c5a1734-680c-472c-a544-723789285116","Type":"ContainerDied","Data":"42e5dd6d121247bc019f1dafb7aef6dd36459a64480cef32c8448c4f97858b10"} Oct 30 19:21:06.358223 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:06.358205 4433 scope.go:122] "RemoveContainer" containerID="42e5dd6d121247bc019f1dafb7aef6dd36459a64480cef32c8448c4f97858b10" Oct 30 19:21:07.363267 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:07.363237 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" event={"ID":"5c5a1734-680c-472c-a544-723789285116","Type":"ContainerStarted","Data":"bbe09223b72cb2ff0dbf4e8bc33af467a50acf326afcc88046bcc91ae86a689d"} Oct 30 19:21:07.363738 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:07.363584 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:21:10.165343 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.165300 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:10.389300 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.389225 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:10.622372 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.622316 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:10.649823 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.649787 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r"] Oct 30 19:21:10.653196 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.653173 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" Oct 30 19:21:10.750435 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.750394 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1db55497-45b8-42dd-99f3-927ed86df4d3-metrics-cert\") pod \"lws-controller-manager-5676f67757-rcj5r\" (UID: \"1db55497-45b8-42dd-99f3-927ed86df4d3\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" Oct 30 19:21:10.750613 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.750453 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1db55497-45b8-42dd-99f3-927ed86df4d3-cert\") pod \"lws-controller-manager-5676f67757-rcj5r\" (UID: \"1db55497-45b8-42dd-99f3-927ed86df4d3\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" Oct 30 19:21:10.750613 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.750540 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1db55497-45b8-42dd-99f3-927ed86df4d3-manager-config\") pod \"lws-controller-manager-5676f67757-rcj5r\" (UID: \"1db55497-45b8-42dd-99f3-927ed86df4d3\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" Oct 30 19:21:10.750717 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.750637 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjsjn\" (UniqueName: \"kubernetes.io/projected/1db55497-45b8-42dd-99f3-927ed86df4d3-kube-api-access-bjsjn\") pod \"lws-controller-manager-5676f67757-rcj5r\" (UID: \"1db55497-45b8-42dd-99f3-927ed86df4d3\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" Oct 30 19:21:10.755163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.755114 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4gpz6\"" Oct 30 19:21:10.755163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.755124 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 19:21:10.755163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.755157 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 19:21:10.755361 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.755160 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:21:10.755361 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.755183 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 19:21:10.755361 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.755279 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 19:21:10.756328 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.756308 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r"] Oct 30 19:21:10.852039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.851995 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1db55497-45b8-42dd-99f3-927ed86df4d3-manager-config\") pod \"lws-controller-manager-5676f67757-rcj5r\" (UID: \"1db55497-45b8-42dd-99f3-927ed86df4d3\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" Oct 30 19:21:10.852247 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.852076 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjsjn\" (UniqueName: \"kubernetes.io/projected/1db55497-45b8-42dd-99f3-927ed86df4d3-kube-api-access-bjsjn\") pod \"lws-controller-manager-5676f67757-rcj5r\" (UID: \"1db55497-45b8-42dd-99f3-927ed86df4d3\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" Oct 30 19:21:10.852247 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.852099 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1db55497-45b8-42dd-99f3-927ed86df4d3-metrics-cert\") pod \"lws-controller-manager-5676f67757-rcj5r\" (UID: \"1db55497-45b8-42dd-99f3-927ed86df4d3\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" Oct 30 19:21:10.852247 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.852124 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1db55497-45b8-42dd-99f3-927ed86df4d3-cert\") pod \"lws-controller-manager-5676f67757-rcj5r\" (UID: \"1db55497-45b8-42dd-99f3-927ed86df4d3\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" Oct 30 19:21:10.852737 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.852712 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1db55497-45b8-42dd-99f3-927ed86df4d3-manager-config\") pod \"lws-controller-manager-5676f67757-rcj5r\" (UID: \"1db55497-45b8-42dd-99f3-927ed86df4d3\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" Oct 30 19:21:10.854567 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.854544 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1db55497-45b8-42dd-99f3-927ed86df4d3-cert\") pod \"lws-controller-manager-5676f67757-rcj5r\" (UID: \"1db55497-45b8-42dd-99f3-927ed86df4d3\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" Oct 30 19:21:10.854660 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.854601 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1db55497-45b8-42dd-99f3-927ed86df4d3-metrics-cert\") pod \"lws-controller-manager-5676f67757-rcj5r\" (UID: \"1db55497-45b8-42dd-99f3-927ed86df4d3\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" Oct 30 19:21:10.912387 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.912311 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjsjn\" (UniqueName: \"kubernetes.io/projected/1db55497-45b8-42dd-99f3-927ed86df4d3-kube-api-access-bjsjn\") pod \"lws-controller-manager-5676f67757-rcj5r\" (UID: \"1db55497-45b8-42dd-99f3-927ed86df4d3\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" Oct 30 19:21:10.964023 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:10.963953 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" Oct 30 19:21:11.333700 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:11.333670 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r"] Oct 30 19:21:11.336603 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:21:11.336573 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db55497_45b8_42dd_99f3_927ed86df4d3.slice/crio-abed14bce8d6ed067f5358fcad65a35199ce3912915070038916d50b98435c89 WatchSource:0}: Error finding container abed14bce8d6ed067f5358fcad65a35199ce3912915070038916d50b98435c89: Status 404 returned error can't find the container with id abed14bce8d6ed067f5358fcad65a35199ce3912915070038916d50b98435c89 Oct 30 19:21:11.380041 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:11.380008 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" event={"ID":"1db55497-45b8-42dd-99f3-927ed86df4d3","Type":"ContainerStarted","Data":"abed14bce8d6ed067f5358fcad65a35199ce3912915070038916d50b98435c89"} Oct 30 19:21:16.401480 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:16.401448 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" event={"ID":"1db55497-45b8-42dd-99f3-927ed86df4d3","Type":"ContainerStarted","Data":"d111b1c7e4e2d7edfbad2f941a86dda88a865e2d7a701ec24cce6a6d1ff881a5"} Oct 30 19:21:16.401885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:16.401579 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" Oct 30 19:21:16.513837 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:16.513777 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" podStartSLOduration=2.066951565 podStartE2EDuration="6.513760895s" totalImagesPullingTime="4.44680933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:16.513148103 +0000 UTC m=+323.447935627" watchObservedRunningTime="2026-10-30 19:21:16.513760895 +0000 UTC m=+323.448548442" Oct 30 19:21:18.370143 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:18.370112 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-6f4hg" Oct 30 19:21:22.070402 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:22.070370 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:22.344877 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:22.344797 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:23.345879 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:23.345849 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:23.370831 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:21:23.370795 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:21:23.567130 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:23.567098 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:24.791632 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:24.791583 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:25.031116 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:25.031085 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:26.017557 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:26.017518 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:27.407240 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:27.407213 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5676f67757-rcj5r" Oct 30 19:21:30.796637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:30.796600 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:31.047481 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.047409 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:31.257761 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.257730 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482"] Oct 30 19:21:31.263108 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.263089 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.306316 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.306243 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b46873-539f-42cb-856a-f28941276775-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.306316 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.306280 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b46873-539f-42cb-856a-f28941276775-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.306316 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.306307 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b46873-539f-42cb-856a-f28941276775-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.306551 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.306424 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b46873-539f-42cb-856a-f28941276775-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.306551 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.306456 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b46873-539f-42cb-856a-f28941276775-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.306551 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.306485 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b46873-539f-42cb-856a-f28941276775-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.306696 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.306565 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b46873-539f-42cb-856a-f28941276775-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.306696 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.306649 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b46873-539f-42cb-856a-f28941276775-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.306696 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.306680 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v7tn\" (UniqueName: \"kubernetes.io/projected/40b46873-539f-42cb-856a-f28941276775-kube-api-access-5v7tn\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.315287 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.315265 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 19:21:31.328822 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.328802 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:21:31.328822 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.328816 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:21:31.364481 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.364462 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-48nh6\"" Oct 30 19:21:31.365955 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.365927 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482"] Oct 30 19:21:31.407563 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.407530 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b46873-539f-42cb-856a-f28941276775-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.407703 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.407577 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b46873-539f-42cb-856a-f28941276775-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.407703 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.407609 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b46873-539f-42cb-856a-f28941276775-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.407703 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.407626 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b46873-539f-42cb-856a-f28941276775-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.407703 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.407669 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b46873-539f-42cb-856a-f28941276775-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.407703 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.407695 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v7tn\" (UniqueName: \"kubernetes.io/projected/40b46873-539f-42cb-856a-f28941276775-kube-api-access-5v7tn\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.407945 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.407774 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b46873-539f-42cb-856a-f28941276775-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.407945 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.407792 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b46873-539f-42cb-856a-f28941276775-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.407945 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.407812 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b46873-539f-42cb-856a-f28941276775-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.408116 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.408096 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b46873-539f-42cb-856a-f28941276775-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.408292 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.408171 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b46873-539f-42cb-856a-f28941276775-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.408402 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.408350 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b46873-539f-42cb-856a-f28941276775-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.408402 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.408365 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b46873-539f-42cb-856a-f28941276775-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.408694 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.408677 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b46873-539f-42cb-856a-f28941276775-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.410333 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.410311 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b46873-539f-42cb-856a-f28941276775-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.410649 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.410630 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b46873-539f-42cb-856a-f28941276775-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.444760 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.444732 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:31.464140 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.464119 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b46873-539f-42cb-856a-f28941276775-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.464613 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.464590 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v7tn\" (UniqueName: \"kubernetes.io/projected/40b46873-539f-42cb-856a-f28941276775-kube-api-access-5v7tn\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5xf482\" (UID: \"40b46873-539f-42cb-856a-f28941276775\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.574036 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.574001 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:31.674996 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.674955 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:31.994913 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:31.994839 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482"] Oct 30 19:21:31.997773 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:21:31.997750 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b46873_539f_42cb_856a_f28941276775.slice/crio-14daa86a04a0cf4780b60462c5540a947eff692b2db0fb4c14dfe7c1802c4ef0 WatchSource:0}: Error finding container 14daa86a04a0cf4780b60462c5540a947eff692b2db0fb4c14dfe7c1802c4ef0: Status 404 returned error can't find the container with id 14daa86a04a0cf4780b60462c5540a947eff692b2db0fb4c14dfe7c1802c4ef0 Oct 30 19:21:32.460339 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:32.460307 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" event={"ID":"40b46873-539f-42cb-856a-f28941276775","Type":"ContainerStarted","Data":"14daa86a04a0cf4780b60462c5540a947eff692b2db0fb4c14dfe7c1802c4ef0"} Oct 30 19:21:32.954518 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:32.954483 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:33.560144 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:33.560107 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:34.209347 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:34.209311 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-299ch"] Oct 30 19:21:34.211352 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:34.211330 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-299ch" Oct 30 19:21:34.263312 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:34.263274 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\"" Oct 30 19:21:34.312603 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:34.312568 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-fw8rx\"" Oct 30 19:21:34.313870 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:34.313846 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-299ch"] Oct 30 19:21:34.331855 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:34.331823 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nqfw\" (UniqueName: \"kubernetes.io/projected/c28650ad-4927-4854-a511-4b8279b97e33-kube-api-access-8nqfw\") pod \"odh-observability-584f99c788-299ch\" (UID: \"c28650ad-4927-4854-a511-4b8279b97e33\") " pod="opendatahub/odh-observability-584f99c788-299ch" Oct 30 19:21:34.332039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:34.331989 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/c28650ad-4927-4854-a511-4b8279b97e33-webhook-certs\") pod \"odh-observability-584f99c788-299ch\" (UID: \"c28650ad-4927-4854-a511-4b8279b97e33\") " pod="opendatahub/odh-observability-584f99c788-299ch" Oct 30 19:21:34.433513 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:34.433464 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/c28650ad-4927-4854-a511-4b8279b97e33-webhook-certs\") pod \"odh-observability-584f99c788-299ch\" (UID: \"c28650ad-4927-4854-a511-4b8279b97e33\") " pod="opendatahub/odh-observability-584f99c788-299ch" Oct 30 19:21:34.433691 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:34.433549 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nqfw\" (UniqueName: \"kubernetes.io/projected/c28650ad-4927-4854-a511-4b8279b97e33-kube-api-access-8nqfw\") pod \"odh-observability-584f99c788-299ch\" (UID: \"c28650ad-4927-4854-a511-4b8279b97e33\") " pod="opendatahub/odh-observability-584f99c788-299ch" Oct 30 19:21:34.433691 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:21:34.433616 4433 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 19:21:34.433796 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:21:34.433698 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c28650ad-4927-4854-a511-4b8279b97e33-webhook-certs podName:c28650ad-4927-4854-a511-4b8279b97e33 nodeName:}" failed. No retries permitted until 2026-10-30 19:21:34.933680637 +0000 UTC m=+341.868468143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/c28650ad-4927-4854-a511-4b8279b97e33-webhook-certs") pod "odh-observability-584f99c788-299ch" (UID: "c28650ad-4927-4854-a511-4b8279b97e33") : secret "odh-observability-webhook-cert" not found Oct 30 19:21:34.497710 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:34.497632 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nqfw\" (UniqueName: \"kubernetes.io/projected/c28650ad-4927-4854-a511-4b8279b97e33-kube-api-access-8nqfw\") pod \"odh-observability-584f99c788-299ch\" (UID: \"c28650ad-4927-4854-a511-4b8279b97e33\") " pod="opendatahub/odh-observability-584f99c788-299ch" Oct 30 19:21:34.598797 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:34.598767 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:34.937066 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:34.937017 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/c28650ad-4927-4854-a511-4b8279b97e33-webhook-certs\") pod \"odh-observability-584f99c788-299ch\" (UID: \"c28650ad-4927-4854-a511-4b8279b97e33\") " pod="opendatahub/odh-observability-584f99c788-299ch" Oct 30 19:21:34.937264 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:21:34.937173 4433 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 19:21:34.937264 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:21:34.937246 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c28650ad-4927-4854-a511-4b8279b97e33-webhook-certs podName:c28650ad-4927-4854-a511-4b8279b97e33 nodeName:}" failed. No retries permitted until 2026-10-30 19:21:35.937224859 +0000 UTC m=+342.872012362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/c28650ad-4927-4854-a511-4b8279b97e33-webhook-certs") pod "odh-observability-584f99c788-299ch" (UID: "c28650ad-4927-4854-a511-4b8279b97e33") : secret "odh-observability-webhook-cert" not found Oct 30 19:21:35.333002 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:35.332939 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:35.582124 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:35.582092 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:21:35.582234 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:35.582163 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:21:35.582234 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:35.582191 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:21:35.946810 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:35.946739 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/c28650ad-4927-4854-a511-4b8279b97e33-webhook-certs\") pod \"odh-observability-584f99c788-299ch\" (UID: \"c28650ad-4927-4854-a511-4b8279b97e33\") " pod="opendatahub/odh-observability-584f99c788-299ch" Oct 30 19:21:35.949066 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:35.949047 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/c28650ad-4927-4854-a511-4b8279b97e33-webhook-certs\") pod \"odh-observability-584f99c788-299ch\" (UID: \"c28650ad-4927-4854-a511-4b8279b97e33\") " pod="opendatahub/odh-observability-584f99c788-299ch" Oct 30 19:21:36.025142 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:36.025107 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-299ch" Oct 30 19:21:36.394865 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:36.394830 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-299ch"] Oct 30 19:21:36.397909 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:21:36.397877 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28650ad_4927_4854_a511_4b8279b97e33.slice/crio-ff1c1adc7a0ec5ff3402c8f5842fa9f36c74dc68daa0ffd0a2a0f5b8bd795bdf WatchSource:0}: Error finding container ff1c1adc7a0ec5ff3402c8f5842fa9f36c74dc68daa0ffd0a2a0f5b8bd795bdf: Status 404 returned error can't find the container with id ff1c1adc7a0ec5ff3402c8f5842fa9f36c74dc68daa0ffd0a2a0f5b8bd795bdf Oct 30 19:21:36.484351 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:36.484314 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" event={"ID":"40b46873-539f-42cb-856a-f28941276775","Type":"ContainerStarted","Data":"5697baa34c3dd4450a7bda32780ac46f9a41a6a86cceb82e6502c00c7181c669"} Oct 30 19:21:36.485369 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:36.485339 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-299ch" event={"ID":"c28650ad-4927-4854-a511-4b8279b97e33","Type":"ContainerStarted","Data":"ff1c1adc7a0ec5ff3402c8f5842fa9f36c74dc68daa0ffd0a2a0f5b8bd795bdf"} Oct 30 19:21:36.574596 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:36.574558 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:36.579080 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:36.579052 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:36.603291 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:36.603237 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" podStartSLOduration=2.021598597 podStartE2EDuration="5.603223398s" totalImagesPullingTime="3.581624801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:36.600545622 +0000 UTC m=+343.535333145" watchObservedRunningTime="2026-10-30 19:21:36.603223398 +0000 UTC m=+343.538010919" Oct 30 19:21:37.489342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:37.489312 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:37.490224 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:37.490202 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5xf482" Oct 30 19:21:39.497835 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:39.497796 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-299ch" event={"ID":"c28650ad-4927-4854-a511-4b8279b97e33","Type":"ContainerStarted","Data":"47d6165608b0b79c5f256083fe0f9570cab8814cf1f4400f62822168c59f96ee"} Oct 30 19:21:39.498248 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:39.498031 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-299ch" Oct 30 19:21:39.612037 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:39.611976 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-299ch" podStartSLOduration=2.633003113 podStartE2EDuration="5.611961741s" totalImagesPullingTime="2.978958628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:39.610853795 +0000 UTC m=+346.545641318" watchObservedRunningTime="2026-10-30 19:21:39.611961741 +0000 UTC m=+346.546749276" Oct 30 19:21:42.728589 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:42.728548 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:43.040071 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:43.039993 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:43.370429 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:43.370394 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:50.504166 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:50.504131 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-299ch" Oct 30 19:21:51.350387 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.350357 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:51.536614 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.536576 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5"] Oct 30 19:21:51.543665 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.543641 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" Oct 30 19:21:51.599849 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.599818 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-f68s7\"" Oct 30 19:21:51.673587 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.673507 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8970bfd-bb0e-4266-92f4-4313f641b671-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-84kt5\" (UID: \"f8970bfd-bb0e-4266-92f4-4313f641b671\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" Oct 30 19:21:51.673587 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.673554 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f8970bfd-bb0e-4266-92f4-4313f641b671-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-84kt5\" (UID: \"f8970bfd-bb0e-4266-92f4-4313f641b671\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" Oct 30 19:21:51.673587 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.673575 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnrxw\" (UniqueName: \"kubernetes.io/projected/f8970bfd-bb0e-4266-92f4-4313f641b671-kube-api-access-mnrxw\") pod \"kserve-module-controller-manager-5cfb77bf8c-84kt5\" (UID: \"f8970bfd-bb0e-4266-92f4-4313f641b671\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" Oct 30 19:21:51.673790 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.673711 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8970bfd-bb0e-4266-92f4-4313f641b671-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-84kt5\" (UID: \"f8970bfd-bb0e-4266-92f4-4313f641b671\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" Oct 30 19:21:51.750760 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.750723 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 19:21:51.752158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.752136 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5"] Oct 30 19:21:51.774395 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.774364 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8970bfd-bb0e-4266-92f4-4313f641b671-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-84kt5\" (UID: \"f8970bfd-bb0e-4266-92f4-4313f641b671\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" Oct 30 19:21:51.774515 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.774416 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f8970bfd-bb0e-4266-92f4-4313f641b671-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-84kt5\" (UID: \"f8970bfd-bb0e-4266-92f4-4313f641b671\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" Oct 30 19:21:51.774515 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.774445 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnrxw\" (UniqueName: \"kubernetes.io/projected/f8970bfd-bb0e-4266-92f4-4313f641b671-kube-api-access-mnrxw\") pod \"kserve-module-controller-manager-5cfb77bf8c-84kt5\" (UID: \"f8970bfd-bb0e-4266-92f4-4313f641b671\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" Oct 30 19:21:51.774515 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.774491 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8970bfd-bb0e-4266-92f4-4313f641b671-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-84kt5\" (UID: \"f8970bfd-bb0e-4266-92f4-4313f641b671\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" Oct 30 19:21:51.774905 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.774887 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f8970bfd-bb0e-4266-92f4-4313f641b671-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-84kt5\" (UID: \"f8970bfd-bb0e-4266-92f4-4313f641b671\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" Oct 30 19:21:51.774962 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.774944 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8970bfd-bb0e-4266-92f4-4313f641b671-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-84kt5\" (UID: \"f8970bfd-bb0e-4266-92f4-4313f641b671\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" Oct 30 19:21:51.775472 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.775455 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8970bfd-bb0e-4266-92f4-4313f641b671-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-84kt5\" (UID: \"f8970bfd-bb0e-4266-92f4-4313f641b671\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" Oct 30 19:21:51.836302 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.836274 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnrxw\" (UniqueName: \"kubernetes.io/projected/f8970bfd-bb0e-4266-92f4-4313f641b671-kube-api-access-mnrxw\") pod \"kserve-module-controller-manager-5cfb77bf8c-84kt5\" (UID: \"f8970bfd-bb0e-4266-92f4-4313f641b671\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" Oct 30 19:21:51.856148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.856123 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" Oct 30 19:21:51.998038 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:51.997914 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:52.276493 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:52.276311 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5"] Oct 30 19:21:52.278959 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:21:52.278929 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8970bfd_bb0e_4266_92f4_4313f641b671.slice/crio-78769c3c9d4d61c5f00902fc27f5a70ae9b686f98ad7c61890e10f4e60061f88 WatchSource:0}: Error finding container 78769c3c9d4d61c5f00902fc27f5a70ae9b686f98ad7c61890e10f4e60061f88: Status 404 returned error can't find the container with id 78769c3c9d4d61c5f00902fc27f5a70ae9b686f98ad7c61890e10f4e60061f88 Oct 30 19:21:52.548845 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:52.548770 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" event={"ID":"f8970bfd-bb0e-4266-92f4-4313f641b671","Type":"ContainerStarted","Data":"78769c3c9d4d61c5f00902fc27f5a70ae9b686f98ad7c61890e10f4e60061f88"} Oct 30 19:21:53.370322 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:21:53.370253 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:21:54.618942 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:54.618863 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:54.871039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:54.870959 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:55.563238 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:55.563199 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" event={"ID":"f8970bfd-bb0e-4266-92f4-4313f641b671","Type":"ContainerStarted","Data":"58a009fd236f9c7fe2994d7d2cd8eda7326e29149128c93a5329a01900365d14"} Oct 30 19:21:55.563442 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:55.563278 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" Oct 30 19:21:55.757249 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:21:55.757197 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" podStartSLOduration=2.1375943570000002 podStartE2EDuration="4.757182328s" totalImagesPullingTime="2.619587971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:55.75659376 +0000 UTC m=+362.691381280" watchObservedRunningTime="2026-10-30 19:21:55.757182328 +0000 UTC m=+362.691969852" Oct 30 19:22:16.568746 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:16.568715 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-84kt5" Oct 30 19:22:23.367048 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:22:23.367013 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:22:48.031132 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:48.031093 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:48.260142 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:48.260064 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:48.617641 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:48.617610 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:48.871924 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:48.871838 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:48.875146 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:48.875123 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-tzbdd"] Oct 30 19:22:48.881798 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:48.881780 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-tzbdd" Oct 30 19:22:48.930953 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:48.930923 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztzn9\" (UniqueName: \"kubernetes.io/projected/2959b203-6fb1-492c-bcbf-c1034896b4cf-kube-api-access-ztzn9\") pod \"kserve-controller-manager-776cc66d47-tzbdd\" (UID: \"2959b203-6fb1-492c-bcbf-c1034896b4cf\") " pod="opendatahub/kserve-controller-manager-776cc66d47-tzbdd" Oct 30 19:22:48.931113 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:48.931062 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2959b203-6fb1-492c-bcbf-c1034896b4cf-cert\") pod \"kserve-controller-manager-776cc66d47-tzbdd\" (UID: \"2959b203-6fb1-492c-bcbf-c1034896b4cf\") " pod="opendatahub/kserve-controller-manager-776cc66d47-tzbdd" Oct 30 19:22:48.937629 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:48.937595 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-tzbdd"] Oct 30 19:22:48.985278 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:48.985247 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 19:22:48.985429 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:48.985355 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-xtrcj\"" Oct 30 19:22:49.031788 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:49.031756 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2959b203-6fb1-492c-bcbf-c1034896b4cf-cert\") pod \"kserve-controller-manager-776cc66d47-tzbdd\" (UID: \"2959b203-6fb1-492c-bcbf-c1034896b4cf\") " pod="opendatahub/kserve-controller-manager-776cc66d47-tzbdd" Oct 30 19:22:49.032189 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:49.031797 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztzn9\" (UniqueName: \"kubernetes.io/projected/2959b203-6fb1-492c-bcbf-c1034896b4cf-kube-api-access-ztzn9\") pod \"kserve-controller-manager-776cc66d47-tzbdd\" (UID: \"2959b203-6fb1-492c-bcbf-c1034896b4cf\") " pod="opendatahub/kserve-controller-manager-776cc66d47-tzbdd" Oct 30 19:22:49.034281 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:49.034253 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2959b203-6fb1-492c-bcbf-c1034896b4cf-cert\") pod \"kserve-controller-manager-776cc66d47-tzbdd\" (UID: \"2959b203-6fb1-492c-bcbf-c1034896b4cf\") " pod="opendatahub/kserve-controller-manager-776cc66d47-tzbdd" Oct 30 19:22:49.088680 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:49.088656 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztzn9\" (UniqueName: \"kubernetes.io/projected/2959b203-6fb1-492c-bcbf-c1034896b4cf-kube-api-access-ztzn9\") pod \"kserve-controller-manager-776cc66d47-tzbdd\" (UID: \"2959b203-6fb1-492c-bcbf-c1034896b4cf\") " pod="opendatahub/kserve-controller-manager-776cc66d47-tzbdd" Oct 30 19:22:49.108060 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:49.108028 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:49.194036 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:49.193947 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-tzbdd" Oct 30 19:22:49.563329 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:49.563302 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-tzbdd"] Oct 30 19:22:49.566414 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:22:49.566387 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2959b203_6fb1_492c_bcbf_c1034896b4cf.slice/crio-3635cd5cd8f533f470481f2625bd843afaf9fa26f6489e551570970e89c4966f WatchSource:0}: Error finding container 3635cd5cd8f533f470481f2625bd843afaf9fa26f6489e551570970e89c4966f: Status 404 returned error can't find the container with id 3635cd5cd8f533f470481f2625bd843afaf9fa26f6489e551570970e89c4966f Oct 30 19:22:49.682430 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:49.682393 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:49.770625 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:49.770594 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-tzbdd" event={"ID":"2959b203-6fb1-492c-bcbf-c1034896b4cf","Type":"ContainerStarted","Data":"3635cd5cd8f533f470481f2625bd843afaf9fa26f6489e551570970e89c4966f"} Oct 30 19:22:50.312081 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:50.312039 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:52.783709 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:52.783678 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-tzbdd" event={"ID":"2959b203-6fb1-492c-bcbf-c1034896b4cf","Type":"ContainerStarted","Data":"3d150fa583306a49a237da19fbe2d67c989a90d13c0d8297aedf13528c586a36"} Oct 30 19:22:52.784077 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:52.783795 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-tzbdd" Oct 30 19:22:52.900197 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:52.900154 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-tzbdd" podStartSLOduration=2.212934688 podStartE2EDuration="4.90014189s" totalImagesPullingTime="2.687207202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:52.898480308 +0000 UTC m=+419.833267836" watchObservedRunningTime="2026-10-30 19:22:52.90014189 +0000 UTC m=+419.834929413" Oct 30 19:22:53.366756 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:22:53.366723 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:22:54.987137 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:54.987105 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:55.466209 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:55.466176 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:55.724841 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:55.724768 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:56.102827 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:56.102796 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:56.403467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:56.403386 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:56.635879 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:56.635851 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:56.870001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:22:56.869944 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:23.364758 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:23:23.364723 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:23:23.792232 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:23:23.792156 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-tzbdd" Oct 30 19:23:50.019847 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:23:50.019815 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:53.365107 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:23:53.365061 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:23:55.283949 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:23:55.283909 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:05.082908 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:05.082825 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:05.305742 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:05.305707 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:23.365284 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:24:23.365248 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:24:24.765994 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:24.765941 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:25.529330 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:25.529290 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:26.331116 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:26.331081 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:29.654461 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:29.654431 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:29.906163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:29.906081 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:30.950020 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:30.949973 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg"] Oct 30 19:24:30.953510 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:30.953490 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" Oct 30 19:24:31.006830 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:31.006801 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bjhkz\"" Oct 30 19:24:31.056389 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:31.056363 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 19:24:31.056389 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:31.056363 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 19:24:31.057792 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:31.057770 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg"] Oct 30 19:24:31.097608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:31.097571 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxkx6\" (UniqueName: \"kubernetes.io/projected/3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5-kube-api-access-bxkx6\") pod \"limitador-operator-controller-manager-856dd584c5-lfpgg\" (UID: \"3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" Oct 30 19:24:31.198602 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:31.198558 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxkx6\" (UniqueName: \"kubernetes.io/projected/3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5-kube-api-access-bxkx6\") pod \"limitador-operator-controller-manager-856dd584c5-lfpgg\" (UID: \"3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" Oct 30 19:24:31.255615 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:31.255542 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxkx6\" (UniqueName: \"kubernetes.io/projected/3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5-kube-api-access-bxkx6\") pod \"limitador-operator-controller-manager-856dd584c5-lfpgg\" (UID: \"3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" Oct 30 19:24:31.266185 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:31.266163 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" Oct 30 19:24:31.643203 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:31.643179 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg"] Oct 30 19:24:31.648443 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:24:31.648415 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0ca3c5_fb47_44b6_8926_3ffb751b1dc5.slice/crio-05e9e7e307249c8e59d76ff5574ab482cc6ddf3c80d50ea1c4ebf4baa36ad378 WatchSource:0}: Error finding container 05e9e7e307249c8e59d76ff5574ab482cc6ddf3c80d50ea1c4ebf4baa36ad378: Status 404 returned error can't find the container with id 05e9e7e307249c8e59d76ff5574ab482cc6ddf3c80d50ea1c4ebf4baa36ad378 Oct 30 19:24:31.750133 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:31.750095 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:31.999613 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:31.999531 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:32.161359 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:32.161322 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" event={"ID":"3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5","Type":"ContainerStarted","Data":"05e9e7e307249c8e59d76ff5574ab482cc6ddf3c80d50ea1c4ebf4baa36ad378"} Oct 30 19:24:33.371787 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:33.371757 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:33.616437 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:33.616397 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:34.151856 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:34.151823 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:35.175703 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:35.175680 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" event={"ID":"3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5","Type":"ContainerStarted","Data":"65113c16eef0e9a61cb9b130cc1c9d91efa542e7ee7b8ee838e9b15a6700edc5"} Oct 30 19:24:35.176031 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:35.175810 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" Oct 30 19:24:35.337604 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:35.337552 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" podStartSLOduration=1.875699317 podStartE2EDuration="5.337538551s" totalImagesPullingTime="3.461839234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:35.336566783 +0000 UTC m=+522.271354298" watchObservedRunningTime="2026-10-30 19:24:35.337538551 +0000 UTC m=+522.272326074" Oct 30 19:24:35.595403 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:35.595370 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:35.836756 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:35.836727 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:39.943335 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:39.943298 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:40.266539 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:40.266468 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:40.500578 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:40.500539 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:40.874913 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:40.874878 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:41.133329 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:41.133246 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:42.734285 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:42.734245 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6"] Oct 30 19:24:42.737712 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:42.737644 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6" Oct 30 19:24:42.789782 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:42.789756 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 19:24:42.800058 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:42.800030 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:42.839215 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:42.839184 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 19:24:42.839215 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:42.839194 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8l6kf\"" Oct 30 19:24:42.840572 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:42.840552 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6"] Oct 30 19:24:42.901068 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:42.901040 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/67eba015-5cff-4ff1-b851-3f53d4497847-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5drm6\" (UID: \"67eba015-5cff-4ff1-b851-3f53d4497847\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6" Oct 30 19:24:42.901187 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:42.901082 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kckt\" (UniqueName: \"kubernetes.io/projected/67eba015-5cff-4ff1-b851-3f53d4497847-kube-api-access-7kckt\") pod \"kuadrant-console-plugin-66c4fb575d-5drm6\" (UID: \"67eba015-5cff-4ff1-b851-3f53d4497847\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6" Oct 30 19:24:42.901224 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:42.901193 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/67eba015-5cff-4ff1-b851-3f53d4497847-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5drm6\" (UID: \"67eba015-5cff-4ff1-b851-3f53d4497847\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6" Oct 30 19:24:43.001866 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:43.001802 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kckt\" (UniqueName: \"kubernetes.io/projected/67eba015-5cff-4ff1-b851-3f53d4497847-kube-api-access-7kckt\") pod \"kuadrant-console-plugin-66c4fb575d-5drm6\" (UID: \"67eba015-5cff-4ff1-b851-3f53d4497847\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6" Oct 30 19:24:43.001866 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:43.001857 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/67eba015-5cff-4ff1-b851-3f53d4497847-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5drm6\" (UID: \"67eba015-5cff-4ff1-b851-3f53d4497847\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6" Oct 30 19:24:43.002060 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:43.001902 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/67eba015-5cff-4ff1-b851-3f53d4497847-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5drm6\" (UID: \"67eba015-5cff-4ff1-b851-3f53d4497847\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6" Oct 30 19:24:43.002464 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:43.002444 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/67eba015-5cff-4ff1-b851-3f53d4497847-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5drm6\" (UID: \"67eba015-5cff-4ff1-b851-3f53d4497847\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6" Oct 30 19:24:43.004385 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:43.004355 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/67eba015-5cff-4ff1-b851-3f53d4497847-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5drm6\" (UID: \"67eba015-5cff-4ff1-b851-3f53d4497847\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6" Oct 30 19:24:43.038381 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:43.038351 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:43.060095 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:43.060070 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kckt\" (UniqueName: \"kubernetes.io/projected/67eba015-5cff-4ff1-b851-3f53d4497847-kube-api-access-7kckt\") pod \"kuadrant-console-plugin-66c4fb575d-5drm6\" (UID: \"67eba015-5cff-4ff1-b851-3f53d4497847\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6" Oct 30 19:24:43.349096 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:43.349059 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6" Oct 30 19:24:43.719872 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:43.719796 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6"] Oct 30 19:24:43.722660 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:24:43.722634 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67eba015_5cff_4ff1_b851_3f53d4497847.slice/crio-9bb3c565b7d461e4b8b58c408acd49810a12bab2914ff3bc88cb216578e376c5 WatchSource:0}: Error finding container 9bb3c565b7d461e4b8b58c408acd49810a12bab2914ff3bc88cb216578e376c5: Status 404 returned error can't find the container with id 9bb3c565b7d461e4b8b58c408acd49810a12bab2914ff3bc88cb216578e376c5 Oct 30 19:24:44.213938 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:44.213895 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6" event={"ID":"67eba015-5cff-4ff1-b851-3f53d4497847","Type":"ContainerStarted","Data":"9bb3c565b7d461e4b8b58c408acd49810a12bab2914ff3bc88cb216578e376c5"} Oct 30 19:24:46.182515 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:46.182485 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" Oct 30 19:24:46.975832 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:46.975805 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:47.443776 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:47.443740 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:49.415466 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:49.415423 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:49.694966 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:49.694883 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:50.421265 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:50.421238 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:51.247460 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:51.247429 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6" event={"ID":"67eba015-5cff-4ff1-b851-3f53d4497847","Type":"ContainerStarted","Data":"7aad86d61021fe2bc4e949d6ce36d1c089edbd5d4354f7068a5a95084b7f29d0"} Oct 30 19:24:51.361527 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:51.361474 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5drm6" podStartSLOduration=2.66523848 podStartE2EDuration="9.361460988s" totalImagesPullingTime="6.696222508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:51.360615092 +0000 UTC m=+538.295402616" watchObservedRunningTime="2026-10-30 19:24:51.361460988 +0000 UTC m=+538.296248512" Oct 30 19:24:53.364580 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:24:53.364549 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:24:55.638678 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:55.638648 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:55.975331 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:55.975256 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:56.324946 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:24:56.324917 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:01.406785 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:01.406751 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:01.657365 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:01.657283 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:01.922367 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:01.922272 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:02.905487 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:02.905453 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg"] Oct 30 19:25:02.905955 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:02.905648 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" podUID="3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5" containerName="manager" containerID="cri-o://65113c16eef0e9a61cb9b130cc1c9d91efa542e7ee7b8ee838e9b15a6700edc5" gracePeriod=2 Oct 30 19:25:02.955041 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:02.955008 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg"] Oct 30 19:25:02.956190 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:02.956166 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-gnbjg"] Oct 30 19:25:02.956523 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:02.956509 4433 cpu_manager.go:414] "containerMap: removing container" podUID="3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5" containerName="manager" Oct 30 19:25:02.956590 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:02.956523 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5" containerName="manager" Oct 30 19:25:02.956590 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:25:02.956549 4433 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-lfpgg\" is forbidden: User \"system:node:ip-10-0-141-5.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-5.us-west-1.compute.internal' and this object" podUID="3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" Oct 30 19:25:02.956590 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:02.956588 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5" containerName="manager" Oct 30 19:25:02.959633 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:02.959613 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-gnbjg" Oct 30 19:25:03.010408 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:25:03.010369 4433 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-lfpgg\" is forbidden: User \"system:node:ip-10-0-141-5.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-5.us-west-1.compute.internal' and this object" podUID="3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" Oct 30 19:25:03.031326 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:03.031281 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-gnbjg"] Oct 30 19:25:03.047800 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:03.047765 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xkxl\" (UniqueName: \"kubernetes.io/projected/766d6d65-943d-4ae0-9c0c-1601453ff404-kube-api-access-8xkxl\") pod \"limitador-operator-controller-manager-856dd584c5-gnbjg\" (UID: \"766d6d65-943d-4ae0-9c0c-1601453ff404\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-gnbjg" Oct 30 19:25:03.148404 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:03.148373 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xkxl\" (UniqueName: \"kubernetes.io/projected/766d6d65-943d-4ae0-9c0c-1601453ff404-kube-api-access-8xkxl\") pod \"limitador-operator-controller-manager-856dd584c5-gnbjg\" (UID: \"766d6d65-943d-4ae0-9c0c-1601453ff404\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-gnbjg" Oct 30 19:25:03.202779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:03.202754 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" Oct 30 19:25:03.208071 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:03.208038 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xkxl\" (UniqueName: \"kubernetes.io/projected/766d6d65-943d-4ae0-9c0c-1601453ff404-kube-api-access-8xkxl\") pod \"limitador-operator-controller-manager-856dd584c5-gnbjg\" (UID: \"766d6d65-943d-4ae0-9c0c-1601453ff404\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-gnbjg" Oct 30 19:25:03.249669 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:03.249640 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5-kube-api-access-bxkx6\" (UniqueName: \"kubernetes.io/projected/3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5-kube-api-access-bxkx6\") pod \"3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5\" (UID: \"3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5\") " Oct 30 19:25:03.251797 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:03.251771 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5-kube-api-access-bxkx6" pod "3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5" (UID: "3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5"). InnerVolumeSpecName "kube-api-access-bxkx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:03.253802 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:25:03.253772 4433 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-lfpgg\" is forbidden: User \"system:node:ip-10-0-141-5.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-5.us-west-1.compute.internal' and this object" podUID="3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" Oct 30 19:25:03.292689 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:03.292658 4433 generic.go:356] "Generic (PLEG): container finished" podID="3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5" containerID="65113c16eef0e9a61cb9b130cc1c9d91efa542e7ee7b8ee838e9b15a6700edc5" exitCode=0 Oct 30 19:25:03.292851 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:03.292706 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" Oct 30 19:25:03.292851 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:03.292716 4433 scope.go:122] "RemoveContainer" containerID="65113c16eef0e9a61cb9b130cc1c9d91efa542e7ee7b8ee838e9b15a6700edc5" Oct 30 19:25:03.297559 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:03.297538 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-gnbjg" Oct 30 19:25:03.301136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:03.301122 4433 scope.go:122] "RemoveContainer" containerID="65113c16eef0e9a61cb9b130cc1c9d91efa542e7ee7b8ee838e9b15a6700edc5" Oct 30 19:25:03.301391 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:25:03.301374 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65113c16eef0e9a61cb9b130cc1c9d91efa542e7ee7b8ee838e9b15a6700edc5\": container with ID starting with 65113c16eef0e9a61cb9b130cc1c9d91efa542e7ee7b8ee838e9b15a6700edc5 not found: ID does not exist" containerID="65113c16eef0e9a61cb9b130cc1c9d91efa542e7ee7b8ee838e9b15a6700edc5" Oct 30 19:25:03.301478 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:03.301400 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65113c16eef0e9a61cb9b130cc1c9d91efa542e7ee7b8ee838e9b15a6700edc5"} err="failed to get container status \"65113c16eef0e9a61cb9b130cc1c9d91efa542e7ee7b8ee838e9b15a6700edc5\": rpc error: code = NotFound desc = could not find container \"65113c16eef0e9a61cb9b130cc1c9d91efa542e7ee7b8ee838e9b15a6700edc5\": container with ID starting with 65113c16eef0e9a61cb9b130cc1c9d91efa542e7ee7b8ee838e9b15a6700edc5 not found: ID does not exist" Oct 30 19:25:03.348239 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:25:03.346958 4433 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-lfpgg\" is forbidden: User \"system:node:ip-10-0-141-5.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-5.us-west-1.compute.internal' and this object" podUID="3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" Oct 30 19:25:03.350993 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:03.350944 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxkx6\" (UniqueName: \"kubernetes.io/projected/3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5-kube-api-access-bxkx6\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:25:03.398424 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:25:03.398386 4433 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-lfpgg\" is forbidden: User \"system:node:ip-10-0-141-5.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-5.us-west-1.compute.internal' and this object" podUID="3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" Oct 30 19:25:03.670164 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:03.670045 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-gnbjg"] Oct 30 19:25:03.672839 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:25:03.672811 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766d6d65_943d_4ae0_9c0c_1601453ff404.slice/crio-cf078426515cea3623314cb3426b729686720197b349e24fc78207c6c9fdec91 WatchSource:0}: Error finding container cf078426515cea3623314cb3426b729686720197b349e24fc78207c6c9fdec91: Status 404 returned error can't find the container with id cf078426515cea3623314cb3426b729686720197b349e24fc78207c6c9fdec91 Oct 30 19:25:04.299004 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:04.298948 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-gnbjg" event={"ID":"766d6d65-943d-4ae0-9c0c-1601453ff404","Type":"ContainerStarted","Data":"7fd5fb418fdd6cb23c0a45a00cb72986dcc40f0f4714dca05394b388105757e8"} Oct 30 19:25:04.299489 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:04.299025 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-gnbjg" event={"ID":"766d6d65-943d-4ae0-9c0c-1601453ff404","Type":"ContainerStarted","Data":"cf078426515cea3623314cb3426b729686720197b349e24fc78207c6c9fdec91"} Oct 30 19:25:04.299489 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:04.299057 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-gnbjg" Oct 30 19:25:04.354024 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:25:04.353970 4433 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-lfpgg\" is forbidden: User \"system:node:ip-10-0-141-5.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-5.us-west-1.compute.internal' and this object" podUID="3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-lfpgg" Oct 30 19:25:05.216573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:05.214269 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5" path="/var/lib/kubelet/pods/3b0ca3c5-fb47-44b6-8926-3ffb751b1dc5/volumes" Oct 30 19:25:05.756307 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:05.756272 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:07.394720 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:07.394683 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:09.583503 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:09.583469 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:09.816092 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:09.816061 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:12.599104 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:12.599063 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:12.853508 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:12.853429 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:15.306044 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:15.306009 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-gnbjg" Oct 30 19:25:15.424434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:15.424380 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-gnbjg" podStartSLOduration=13.42436725 podStartE2EDuration="13.42436725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:04.481952492 +0000 UTC m=+551.416740016" watchObservedRunningTime="2026-10-30 19:25:15.42436725 +0000 UTC m=+562.359154773" Oct 30 19:25:18.405781 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:18.405739 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:21.664060 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:21.664023 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:23.365686 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:25:23.365651 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:25:29.412325 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:29.412290 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:29.718642 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:29.718565 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:35.564854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:35.564746 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:35.788590 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:35.788553 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:42.935884 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:42.935851 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:50.708661 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:50.708624 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:53.365483 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:25:53.365444 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:25:54.059425 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:54.059391 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:54.376425 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:54.376394 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:55.962847 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:25:55.962812 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:02.895330 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:02.895291 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:03.281368 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:03.281289 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:03.505554 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:03.505522 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:04.372517 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:04.372477 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ckvrp"] Oct 30 19:26:04.377111 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:04.377086 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ckvrp" Oct 30 19:26:04.430097 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:04.430069 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-g6bkn\"" Oct 30 19:26:04.431326 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:04.431306 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ckvrp"] Oct 30 19:26:04.448510 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:04.448488 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb4p2\" (UniqueName: \"kubernetes.io/projected/3baa2798-db38-4f33-8019-92687f2a4d5e-kube-api-access-lb4p2\") pod \"authorino-57f7f8659f-ckvrp\" (UID: \"3baa2798-db38-4f33-8019-92687f2a4d5e\") " pod="kuadrant-system/authorino-57f7f8659f-ckvrp" Oct 30 19:26:04.549271 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:04.549241 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb4p2\" (UniqueName: \"kubernetes.io/projected/3baa2798-db38-4f33-8019-92687f2a4d5e-kube-api-access-lb4p2\") pod \"authorino-57f7f8659f-ckvrp\" (UID: \"3baa2798-db38-4f33-8019-92687f2a4d5e\") " pod="kuadrant-system/authorino-57f7f8659f-ckvrp" Oct 30 19:26:04.605389 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:04.605364 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb4p2\" (UniqueName: \"kubernetes.io/projected/3baa2798-db38-4f33-8019-92687f2a4d5e-kube-api-access-lb4p2\") pod \"authorino-57f7f8659f-ckvrp\" (UID: \"3baa2798-db38-4f33-8019-92687f2a4d5e\") " pod="kuadrant-system/authorino-57f7f8659f-ckvrp" Oct 30 19:26:04.609116 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:04.609090 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:04.688393 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:04.688328 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ckvrp" Oct 30 19:26:04.842476 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:04.842437 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:05.061432 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:05.061362 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ckvrp"] Oct 30 19:26:05.064059 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:26:05.064032 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3baa2798_db38_4f33_8019_92687f2a4d5e.slice/crio-a1d0478b49623e9afe10d4dde75f2d4d9fc4aaae431181e86d4e9847ec8bbd5a WatchSource:0}: Error finding container a1d0478b49623e9afe10d4dde75f2d4d9fc4aaae431181e86d4e9847ec8bbd5a: Status 404 returned error can't find the container with id a1d0478b49623e9afe10d4dde75f2d4d9fc4aaae431181e86d4e9847ec8bbd5a Oct 30 19:26:05.065346 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:05.065330 4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:26:05.546747 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:05.546714 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ckvrp" event={"ID":"3baa2798-db38-4f33-8019-92687f2a4d5e","Type":"ContainerStarted","Data":"a1d0478b49623e9afe10d4dde75f2d4d9fc4aaae431181e86d4e9847ec8bbd5a"} Oct 30 19:26:05.961651 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:05.961616 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:06.934476 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:06.934447 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-str7s"] Oct 30 19:26:06.938757 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:06.938738 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-str7s" Oct 30 19:26:06.972345 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:06.972308 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddztn\" (UniqueName: \"kubernetes.io/projected/7768d06b-2aaf-4226-b162-3228bf2a7459-kube-api-access-ddztn\") pod \"authorino-56c6c9ddd5-str7s\" (UID: \"7768d06b-2aaf-4226-b162-3228bf2a7459\") " pod="kuadrant-system/authorino-56c6c9ddd5-str7s" Oct 30 19:26:06.995065 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:06.995044 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-str7s"] Oct 30 19:26:07.073357 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:07.073326 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddztn\" (UniqueName: \"kubernetes.io/projected/7768d06b-2aaf-4226-b162-3228bf2a7459-kube-api-access-ddztn\") pod \"authorino-56c6c9ddd5-str7s\" (UID: \"7768d06b-2aaf-4226-b162-3228bf2a7459\") " pod="kuadrant-system/authorino-56c6c9ddd5-str7s" Oct 30 19:26:07.130505 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:07.130473 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddztn\" (UniqueName: \"kubernetes.io/projected/7768d06b-2aaf-4226-b162-3228bf2a7459-kube-api-access-ddztn\") pod \"authorino-56c6c9ddd5-str7s\" (UID: \"7768d06b-2aaf-4226-b162-3228bf2a7459\") " pod="kuadrant-system/authorino-56c6c9ddd5-str7s" Oct 30 19:26:07.249992 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:07.249905 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-str7s" Oct 30 19:26:07.624835 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:07.624782 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-str7s"] Oct 30 19:26:07.627715 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:26:07.627688 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7768d06b_2aaf_4226_b162_3228bf2a7459.slice/crio-2bd2a2c348ff3bfb2b16e61ad7c64cc89137899fb5cde096e77688f141dd37a1 WatchSource:0}: Error finding container 2bd2a2c348ff3bfb2b16e61ad7c64cc89137899fb5cde096e77688f141dd37a1: Status 404 returned error can't find the container with id 2bd2a2c348ff3bfb2b16e61ad7c64cc89137899fb5cde096e77688f141dd37a1 Oct 30 19:26:08.561193 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:08.561151 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-str7s" event={"ID":"7768d06b-2aaf-4226-b162-3228bf2a7459","Type":"ContainerStarted","Data":"2bd2a2c348ff3bfb2b16e61ad7c64cc89137899fb5cde096e77688f141dd37a1"} Oct 30 19:26:08.562832 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:08.562798 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ckvrp" event={"ID":"3baa2798-db38-4f33-8019-92687f2a4d5e","Type":"ContainerStarted","Data":"f1d0bdf06efeb04ca570015715d5a4d7a34538bc899905807fbbad621c19d208"} Oct 30 19:26:08.674626 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:08.674580 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-ckvrp" podStartSLOduration=1.727177947 podStartE2EDuration="4.67456594s" totalImagesPullingTime="2.947387993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:08.67375148 +0000 UTC m=+615.608539002" watchObservedRunningTime="2026-10-30 19:26:08.67456594 +0000 UTC m=+615.609353465" Oct 30 19:26:09.568583 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:09.568543 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-str7s" event={"ID":"7768d06b-2aaf-4226-b162-3228bf2a7459","Type":"ContainerStarted","Data":"e1e487e2f6eb6cccdee6aa8bae46bcfbf6448518ebb6011660e5680f59fc70df"} Oct 30 19:26:09.680024 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:09.679948 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-str7s" podStartSLOduration=2.416969601 podStartE2EDuration="3.679931986s" totalImagesPullingTime="1.262962385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:09.679585329 +0000 UTC m=+616.614372852" watchObservedRunningTime="2026-10-30 19:26:09.679931986 +0000 UTC m=+616.614719509" Oct 30 19:26:09.730173 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:09.730140 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ckvrp"] Oct 30 19:26:10.572566 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:10.572526 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-ckvrp" podUID="3baa2798-db38-4f33-8019-92687f2a4d5e" containerName="authorino" containerID="cri-o://f1d0bdf06efeb04ca570015715d5a4d7a34538bc899905807fbbad621c19d208" gracePeriod=30 Oct 30 19:26:10.829769 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:10.829708 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ckvrp" Oct 30 19:26:10.905775 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:10.905740 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3baa2798-db38-4f33-8019-92687f2a4d5e-kube-api-access-lb4p2\" (UniqueName: \"kubernetes.io/projected/3baa2798-db38-4f33-8019-92687f2a4d5e-kube-api-access-lb4p2\") pod \"3baa2798-db38-4f33-8019-92687f2a4d5e\" (UID: \"3baa2798-db38-4f33-8019-92687f2a4d5e\") " Oct 30 19:26:10.907778 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:10.907751 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3baa2798-db38-4f33-8019-92687f2a4d5e-kube-api-access-lb4p2" pod "3baa2798-db38-4f33-8019-92687f2a4d5e" (UID: "3baa2798-db38-4f33-8019-92687f2a4d5e"). InnerVolumeSpecName "kube-api-access-lb4p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:11.007274 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:11.007239 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb4p2\" (UniqueName: \"kubernetes.io/projected/3baa2798-db38-4f33-8019-92687f2a4d5e-kube-api-access-lb4p2\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:26:11.577299 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:11.577262 4433 generic.go:356] "Generic (PLEG): container finished" podID="3baa2798-db38-4f33-8019-92687f2a4d5e" containerID="f1d0bdf06efeb04ca570015715d5a4d7a34538bc899905807fbbad621c19d208" exitCode=0 Oct 30 19:26:11.577762 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:11.577309 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ckvrp" Oct 30 19:26:11.577762 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:11.577343 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ckvrp" event={"ID":"3baa2798-db38-4f33-8019-92687f2a4d5e","Type":"ContainerDied","Data":"f1d0bdf06efeb04ca570015715d5a4d7a34538bc899905807fbbad621c19d208"} Oct 30 19:26:11.577762 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:11.577380 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ckvrp" event={"ID":"3baa2798-db38-4f33-8019-92687f2a4d5e","Type":"ContainerDied","Data":"a1d0478b49623e9afe10d4dde75f2d4d9fc4aaae431181e86d4e9847ec8bbd5a"} Oct 30 19:26:11.577762 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:11.577396 4433 scope.go:122] "RemoveContainer" containerID="f1d0bdf06efeb04ca570015715d5a4d7a34538bc899905807fbbad621c19d208" Oct 30 19:26:11.585206 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:11.585189 4433 scope.go:122] "RemoveContainer" containerID="f1d0bdf06efeb04ca570015715d5a4d7a34538bc899905807fbbad621c19d208" Oct 30 19:26:11.585488 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:26:11.585469 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d0bdf06efeb04ca570015715d5a4d7a34538bc899905807fbbad621c19d208\": container with ID starting with f1d0bdf06efeb04ca570015715d5a4d7a34538bc899905807fbbad621c19d208 not found: ID does not exist" containerID="f1d0bdf06efeb04ca570015715d5a4d7a34538bc899905807fbbad621c19d208" Oct 30 19:26:11.585555 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:11.585497 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d0bdf06efeb04ca570015715d5a4d7a34538bc899905807fbbad621c19d208"} err="failed to get container status \"f1d0bdf06efeb04ca570015715d5a4d7a34538bc899905807fbbad621c19d208\": rpc error: code = NotFound desc = could not find container \"f1d0bdf06efeb04ca570015715d5a4d7a34538bc899905807fbbad621c19d208\": container with ID starting with f1d0bdf06efeb04ca570015715d5a4d7a34538bc899905807fbbad621c19d208 not found: ID does not exist" Oct 30 19:26:11.738530 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:11.738493 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ckvrp"] Oct 30 19:26:11.743911 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:11.743887 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ckvrp"] Oct 30 19:26:13.211023 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:13.210994 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3baa2798-db38-4f33-8019-92687f2a4d5e" path="/var/lib/kubelet/pods/3baa2798-db38-4f33-8019-92687f2a4d5e/volumes" Oct 30 19:26:23.365173 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:26:23.365128 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:26:28.086540 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:28.086501 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:28.335112 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:28.335082 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:38.850677 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:38.850642 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:39.170016 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:39.169919 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:39.389991 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:39.389963 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:51.033675 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:51.033638 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:53.366396 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:26:53.366361 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:26:56.288046 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:56.288004 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:56.826370 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:56.826334 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:57.049158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.049123 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:57.272573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.272489 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:57.455123 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.455082 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9w86q"] Oct 30 19:26:57.455540 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.455442 4433 cpu_manager.go:414] "containerMap: removing container" podUID="3baa2798-db38-4f33-8019-92687f2a4d5e" containerName="authorino" Oct 30 19:26:57.455540 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.455456 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3baa2798-db38-4f33-8019-92687f2a4d5e" containerName="authorino" Oct 30 19:26:57.455540 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.455527 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3baa2798-db38-4f33-8019-92687f2a4d5e" containerName="authorino" Oct 30 19:26:57.459830 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.459806 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9w86q" Oct 30 19:26:57.484110 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.484080 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7btq\" (UniqueName: \"kubernetes.io/projected/b20c1fcc-acca-448e-a906-63bcf1b9a07f-kube-api-access-b7btq\") pod \"authorino-6f6944bcdd-9w86q\" (UID: \"b20c1fcc-acca-448e-a906-63bcf1b9a07f\") " pod="kuadrant-system/authorino-6f6944bcdd-9w86q" Oct 30 19:26:57.502907 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.502881 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:57.514522 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.514495 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9w86q"] Oct 30 19:26:57.584909 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.584874 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7btq\" (UniqueName: \"kubernetes.io/projected/b20c1fcc-acca-448e-a906-63bcf1b9a07f-kube-api-access-b7btq\") pod \"authorino-6f6944bcdd-9w86q\" (UID: \"b20c1fcc-acca-448e-a906-63bcf1b9a07f\") " pod="kuadrant-system/authorino-6f6944bcdd-9w86q" Oct 30 19:26:57.642021 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.641975 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7btq\" (UniqueName: \"kubernetes.io/projected/b20c1fcc-acca-448e-a906-63bcf1b9a07f-kube-api-access-b7btq\") pod \"authorino-6f6944bcdd-9w86q\" (UID: \"b20c1fcc-acca-448e-a906-63bcf1b9a07f\") " pod="kuadrant-system/authorino-6f6944bcdd-9w86q" Oct 30 19:26:57.770353 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.770320 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9w86q" Oct 30 19:26:57.810829 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.810794 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9w86q"] Oct 30 19:26:57.839391 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.839305 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57d457b6df-hg7m5"] Oct 30 19:26:57.842987 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.842951 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57d457b6df-hg7m5" Oct 30 19:26:57.887441 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.887401 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r68jq\" (UniqueName: \"kubernetes.io/projected/8df6052a-1078-4c61-826f-7ce6d7ac0811-kube-api-access-r68jq\") pod \"authorino-57d457b6df-hg7m5\" (UID: \"8df6052a-1078-4c61-826f-7ce6d7ac0811\") " pod="kuadrant-system/authorino-57d457b6df-hg7m5" Oct 30 19:26:57.898272 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.898238 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57d457b6df-hg7m5"] Oct 30 19:26:57.988352 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:57.988313 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68jq\" (UniqueName: \"kubernetes.io/projected/8df6052a-1078-4c61-826f-7ce6d7ac0811-kube-api-access-r68jq\") pod \"authorino-57d457b6df-hg7m5\" (UID: \"8df6052a-1078-4c61-826f-7ce6d7ac0811\") " pod="kuadrant-system/authorino-57d457b6df-hg7m5" Oct 30 19:26:58.045387 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.045353 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r68jq\" (UniqueName: \"kubernetes.io/projected/8df6052a-1078-4c61-826f-7ce6d7ac0811-kube-api-access-r68jq\") pod \"authorino-57d457b6df-hg7m5\" (UID: \"8df6052a-1078-4c61-826f-7ce6d7ac0811\") " pod="kuadrant-system/authorino-57d457b6df-hg7m5" Oct 30 19:26:58.139144 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.139062 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9w86q"] Oct 30 19:26:58.142762 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:26:58.142737 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb20c1fcc_acca_448e_a906_63bcf1b9a07f.slice/crio-eb839b7c6fe1f2b5da0a66d59715c3ad3c6364c20213bbb0fc9db6cecd886559 WatchSource:0}: Error finding container eb839b7c6fe1f2b5da0a66d59715c3ad3c6364c20213bbb0fc9db6cecd886559: Status 404 returned error can't find the container with id eb839b7c6fe1f2b5da0a66d59715c3ad3c6364c20213bbb0fc9db6cecd886559 Oct 30 19:26:58.157436 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.157407 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57d457b6df-hg7m5" Oct 30 19:26:58.247284 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.247249 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57d457b6df-hg7m5"] Oct 30 19:26:58.278870 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.278836 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f8947ddb7-8kgx9"] Oct 30 19:26:58.282311 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.282289 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f8947ddb7-8kgx9" Oct 30 19:26:58.290833 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.290809 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klc6g\" (UniqueName: \"kubernetes.io/projected/2601bf00-9912-4a32-b570-6d5d1a47bec4-kube-api-access-klc6g\") pod \"authorino-6f8947ddb7-8kgx9\" (UID: \"2601bf00-9912-4a32-b570-6d5d1a47bec4\") " pod="kuadrant-system/authorino-6f8947ddb7-8kgx9" Oct 30 19:26:58.290952 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.290848 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2601bf00-9912-4a32-b570-6d5d1a47bec4-tls-cert\") pod \"authorino-6f8947ddb7-8kgx9\" (UID: \"2601bf00-9912-4a32-b570-6d5d1a47bec4\") " pod="kuadrant-system/authorino-6f8947ddb7-8kgx9" Oct 30 19:26:58.337129 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.337097 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 19:26:58.338747 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.338717 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f8947ddb7-8kgx9"] Oct 30 19:26:58.391663 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.391565 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klc6g\" (UniqueName: \"kubernetes.io/projected/2601bf00-9912-4a32-b570-6d5d1a47bec4-kube-api-access-klc6g\") pod \"authorino-6f8947ddb7-8kgx9\" (UID: \"2601bf00-9912-4a32-b570-6d5d1a47bec4\") " pod="kuadrant-system/authorino-6f8947ddb7-8kgx9" Oct 30 19:26:58.391663 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.391614 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2601bf00-9912-4a32-b570-6d5d1a47bec4-tls-cert\") pod \"authorino-6f8947ddb7-8kgx9\" (UID: \"2601bf00-9912-4a32-b570-6d5d1a47bec4\") " pod="kuadrant-system/authorino-6f8947ddb7-8kgx9" Oct 30 19:26:58.394095 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.394070 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2601bf00-9912-4a32-b570-6d5d1a47bec4-tls-cert\") pod \"authorino-6f8947ddb7-8kgx9\" (UID: \"2601bf00-9912-4a32-b570-6d5d1a47bec4\") " pod="kuadrant-system/authorino-6f8947ddb7-8kgx9" Oct 30 19:26:58.448487 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.448457 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klc6g\" (UniqueName: \"kubernetes.io/projected/2601bf00-9912-4a32-b570-6d5d1a47bec4-kube-api-access-klc6g\") pod \"authorino-6f8947ddb7-8kgx9\" (UID: \"2601bf00-9912-4a32-b570-6d5d1a47bec4\") " pod="kuadrant-system/authorino-6f8947ddb7-8kgx9" Oct 30 19:26:58.525021 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.524971 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57d457b6df-hg7m5"] Oct 30 19:26:58.528190 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:26:58.528156 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8df6052a_1078_4c61_826f_7ce6d7ac0811.slice/crio-c8fa6ba8922cce609cf6628fab7f345f7d7998c84ff0e79d3783c9a38139abd2 WatchSource:0}: Error finding container c8fa6ba8922cce609cf6628fab7f345f7d7998c84ff0e79d3783c9a38139abd2: Status 404 returned error can't find the container with id c8fa6ba8922cce609cf6628fab7f345f7d7998c84ff0e79d3783c9a38139abd2 Oct 30 19:26:58.593476 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.593442 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f8947ddb7-8kgx9" Oct 30 19:26:58.777680 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.777645 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57d457b6df-hg7m5" event={"ID":"8df6052a-1078-4c61-826f-7ce6d7ac0811","Type":"ContainerStarted","Data":"c8fa6ba8922cce609cf6628fab7f345f7d7998c84ff0e79d3783c9a38139abd2"} Oct 30 19:26:58.778741 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.778717 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-9w86q" event={"ID":"b20c1fcc-acca-448e-a906-63bcf1b9a07f","Type":"ContainerStarted","Data":"eb839b7c6fe1f2b5da0a66d59715c3ad3c6364c20213bbb0fc9db6cecd886559"} Oct 30 19:26:58.962531 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:58.962446 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f8947ddb7-8kgx9"] Oct 30 19:26:58.965644 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:26:58.965613 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2601bf00_9912_4a32_b570_6d5d1a47bec4.slice/crio-383ee453bf6d559e6dbb9ef46dd33939f6667d78fb2737d6a33685053329ab6d WatchSource:0}: Error finding container 383ee453bf6d559e6dbb9ef46dd33939f6667d78fb2737d6a33685053329ab6d: Status 404 returned error can't find the container with id 383ee453bf6d559e6dbb9ef46dd33939f6667d78fb2737d6a33685053329ab6d Oct 30 19:26:59.783669 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:59.783577 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f8947ddb7-8kgx9" event={"ID":"2601bf00-9912-4a32-b570-6d5d1a47bec4","Type":"ContainerStarted","Data":"383ee453bf6d559e6dbb9ef46dd33939f6667d78fb2737d6a33685053329ab6d"} Oct 30 19:26:59.784824 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:59.784799 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-9w86q" event={"ID":"b20c1fcc-acca-448e-a906-63bcf1b9a07f","Type":"ContainerStarted","Data":"e252bbf1b194825acfefa05d7d11a58e15b8081418ceeefd638bc2298ad6e07e"} Oct 30 19:26:59.784938 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:59.784894 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-9w86q" podUID="b20c1fcc-acca-448e-a906-63bcf1b9a07f" containerName="authorino" containerID="cri-o://e252bbf1b194825acfefa05d7d11a58e15b8081418ceeefd638bc2298ad6e07e" gracePeriod=30 Oct 30 19:26:59.908723 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:26:59.908663 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-9w86q" podStartSLOduration=1.594099607 podStartE2EDuration="2.908643104s" totalImagesPullingTime="1.314543497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:59.908358959 +0000 UTC m=+666.843146483" watchObservedRunningTime="2026-10-30 19:26:59.908643104 +0000 UTC m=+666.843430630" Oct 30 19:27:00.032176 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:00.032148 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9w86q" Oct 30 19:27:00.113091 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:00.113049 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20c1fcc-acca-448e-a906-63bcf1b9a07f-kube-api-access-b7btq\" (UniqueName: \"kubernetes.io/projected/b20c1fcc-acca-448e-a906-63bcf1b9a07f-kube-api-access-b7btq\") pod \"b20c1fcc-acca-448e-a906-63bcf1b9a07f\" (UID: \"b20c1fcc-acca-448e-a906-63bcf1b9a07f\") " Oct 30 19:27:00.115289 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:00.115255 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20c1fcc-acca-448e-a906-63bcf1b9a07f-kube-api-access-b7btq" pod "b20c1fcc-acca-448e-a906-63bcf1b9a07f" (UID: "b20c1fcc-acca-448e-a906-63bcf1b9a07f"). InnerVolumeSpecName "kube-api-access-b7btq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:00.214232 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:00.214196 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7btq\" (UniqueName: \"kubernetes.io/projected/b20c1fcc-acca-448e-a906-63bcf1b9a07f-kube-api-access-b7btq\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:27:00.790502 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:00.790466 4433 generic.go:356] "Generic (PLEG): container finished" podID="b20c1fcc-acca-448e-a906-63bcf1b9a07f" containerID="e252bbf1b194825acfefa05d7d11a58e15b8081418ceeefd638bc2298ad6e07e" exitCode=0 Oct 30 19:27:00.791014 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:00.790510 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9w86q" Oct 30 19:27:00.791014 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:00.790552 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-9w86q" event={"ID":"b20c1fcc-acca-448e-a906-63bcf1b9a07f","Type":"ContainerDied","Data":"e252bbf1b194825acfefa05d7d11a58e15b8081418ceeefd638bc2298ad6e07e"} Oct 30 19:27:00.791014 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:00.790600 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-9w86q" event={"ID":"b20c1fcc-acca-448e-a906-63bcf1b9a07f","Type":"ContainerDied","Data":"eb839b7c6fe1f2b5da0a66d59715c3ad3c6364c20213bbb0fc9db6cecd886559"} Oct 30 19:27:00.791014 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:00.790625 4433 scope.go:122] "RemoveContainer" containerID="e252bbf1b194825acfefa05d7d11a58e15b8081418ceeefd638bc2298ad6e07e" Oct 30 19:27:00.792609 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:00.792466 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57d457b6df-hg7m5" event={"ID":"8df6052a-1078-4c61-826f-7ce6d7ac0811","Type":"ContainerStarted","Data":"500575ebbf34c7b7126618dcc0a1214617a9a693f6f466be551cd1a208911d4a"} Oct 30 19:27:00.792609 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:00.792525 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57d457b6df-hg7m5" podUID="8df6052a-1078-4c61-826f-7ce6d7ac0811" containerName="authorino" containerID="cri-o://500575ebbf34c7b7126618dcc0a1214617a9a693f6f466be551cd1a208911d4a" gracePeriod=30 Oct 30 19:27:00.794929 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:00.794901 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f8947ddb7-8kgx9" event={"ID":"2601bf00-9912-4a32-b570-6d5d1a47bec4","Type":"ContainerStarted","Data":"6a25d1ea34d7d60c0a68f5d5a453f1f123d368cccc8e34219c6c56bf7093cb23"} Oct 30 19:27:00.868519 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:00.868491 4433 scope.go:122] "RemoveContainer" containerID="e252bbf1b194825acfefa05d7d11a58e15b8081418ceeefd638bc2298ad6e07e" Oct 30 19:27:00.868882 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:27:00.868856 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e252bbf1b194825acfefa05d7d11a58e15b8081418ceeefd638bc2298ad6e07e\": container with ID starting with e252bbf1b194825acfefa05d7d11a58e15b8081418ceeefd638bc2298ad6e07e not found: ID does not exist" containerID="e252bbf1b194825acfefa05d7d11a58e15b8081418ceeefd638bc2298ad6e07e" Oct 30 19:27:00.868938 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:00.868894 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e252bbf1b194825acfefa05d7d11a58e15b8081418ceeefd638bc2298ad6e07e"} err="failed to get container status \"e252bbf1b194825acfefa05d7d11a58e15b8081418ceeefd638bc2298ad6e07e\": rpc error: code = NotFound desc = could not find container \"e252bbf1b194825acfefa05d7d11a58e15b8081418ceeefd638bc2298ad6e07e\": container with ID starting with e252bbf1b194825acfefa05d7d11a58e15b8081418ceeefd638bc2298ad6e07e not found: ID does not exist" Oct 30 19:27:00.957040 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:00.956970 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57d457b6df-hg7m5" podStartSLOduration=2.633795852 podStartE2EDuration="3.956955451s" totalImagesPullingTime="1.323159599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:00.956041891 +0000 UTC m=+667.890829415" watchObservedRunningTime="2026-10-30 19:27:00.956955451 +0000 UTC m=+667.891742976" Oct 30 19:27:01.034048 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.034022 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57d457b6df-hg7m5" Oct 30 19:27:01.069414 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.069366 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f8947ddb7-8kgx9" podStartSLOduration=1.793782019 podStartE2EDuration="3.069350676s" totalImagesPullingTime="1.275568657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:01.067940844 +0000 UTC m=+668.002728370" watchObservedRunningTime="2026-10-30 19:27:01.069350676 +0000 UTC m=+668.004138244" Oct 30 19:27:01.094424 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.094388 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-str7s"] Oct 30 19:27:01.094606 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.094585 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-str7s" podUID="7768d06b-2aaf-4226-b162-3228bf2a7459" containerName="authorino" containerID="cri-o://e1e487e2f6eb6cccdee6aa8bae46bcfbf6448518ebb6011660e5680f59fc70df" gracePeriod=30 Oct 30 19:27:01.120768 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.120741 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df6052a-1078-4c61-826f-7ce6d7ac0811-kube-api-access-r68jq\" (UniqueName: \"kubernetes.io/projected/8df6052a-1078-4c61-826f-7ce6d7ac0811-kube-api-access-r68jq\") pod \"8df6052a-1078-4c61-826f-7ce6d7ac0811\" (UID: \"8df6052a-1078-4c61-826f-7ce6d7ac0811\") " Oct 30 19:27:01.122780 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.122750 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df6052a-1078-4c61-826f-7ce6d7ac0811-kube-api-access-r68jq" pod "8df6052a-1078-4c61-826f-7ce6d7ac0811" (UID: "8df6052a-1078-4c61-826f-7ce6d7ac0811"). InnerVolumeSpecName "kube-api-access-r68jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:01.221818 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.221788 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r68jq\" (UniqueName: \"kubernetes.io/projected/8df6052a-1078-4c61-826f-7ce6d7ac0811-kube-api-access-r68jq\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:27:01.290237 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.290205 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9w86q"] Oct 30 19:27:01.291603 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.291577 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9w86q"] Oct 30 19:27:01.330657 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.330634 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-str7s" Oct 30 19:27:01.423781 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.423743 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7768d06b-2aaf-4226-b162-3228bf2a7459-kube-api-access-ddztn\" (UniqueName: \"kubernetes.io/projected/7768d06b-2aaf-4226-b162-3228bf2a7459-kube-api-access-ddztn\") pod \"7768d06b-2aaf-4226-b162-3228bf2a7459\" (UID: \"7768d06b-2aaf-4226-b162-3228bf2a7459\") " Oct 30 19:27:01.425830 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.425800 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7768d06b-2aaf-4226-b162-3228bf2a7459-kube-api-access-ddztn" pod "7768d06b-2aaf-4226-b162-3228bf2a7459" (UID: "7768d06b-2aaf-4226-b162-3228bf2a7459"). InnerVolumeSpecName "kube-api-access-ddztn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:01.525023 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.524991 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddztn\" (UniqueName: \"kubernetes.io/projected/7768d06b-2aaf-4226-b162-3228bf2a7459-kube-api-access-ddztn\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:27:01.802283 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.802247 4433 generic.go:356] "Generic (PLEG): container finished" podID="7768d06b-2aaf-4226-b162-3228bf2a7459" containerID="e1e487e2f6eb6cccdee6aa8bae46bcfbf6448518ebb6011660e5680f59fc70df" exitCode=0 Oct 30 19:27:01.802732 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.802290 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-str7s" Oct 30 19:27:01.802732 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.802347 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-str7s" event={"ID":"7768d06b-2aaf-4226-b162-3228bf2a7459","Type":"ContainerDied","Data":"e1e487e2f6eb6cccdee6aa8bae46bcfbf6448518ebb6011660e5680f59fc70df"} Oct 30 19:27:01.802732 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.802390 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-str7s" event={"ID":"7768d06b-2aaf-4226-b162-3228bf2a7459","Type":"ContainerDied","Data":"2bd2a2c348ff3bfb2b16e61ad7c64cc89137899fb5cde096e77688f141dd37a1"} Oct 30 19:27:01.802732 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.802413 4433 scope.go:122] "RemoveContainer" containerID="e1e487e2f6eb6cccdee6aa8bae46bcfbf6448518ebb6011660e5680f59fc70df" Oct 30 19:27:01.803789 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.803768 4433 generic.go:356] "Generic (PLEG): container finished" podID="8df6052a-1078-4c61-826f-7ce6d7ac0811" containerID="500575ebbf34c7b7126618dcc0a1214617a9a693f6f466be551cd1a208911d4a" exitCode=0 Oct 30 19:27:01.803838 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.803825 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57d457b6df-hg7m5" Oct 30 19:27:01.803887 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.803858 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57d457b6df-hg7m5" event={"ID":"8df6052a-1078-4c61-826f-7ce6d7ac0811","Type":"ContainerDied","Data":"500575ebbf34c7b7126618dcc0a1214617a9a693f6f466be551cd1a208911d4a"} Oct 30 19:27:01.803950 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.803889 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57d457b6df-hg7m5" event={"ID":"8df6052a-1078-4c61-826f-7ce6d7ac0811","Type":"ContainerDied","Data":"c8fa6ba8922cce609cf6628fab7f345f7d7998c84ff0e79d3783c9a38139abd2"} Oct 30 19:27:01.811198 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.811180 4433 scope.go:122] "RemoveContainer" containerID="e1e487e2f6eb6cccdee6aa8bae46bcfbf6448518ebb6011660e5680f59fc70df" Oct 30 19:27:01.811452 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:27:01.811433 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e487e2f6eb6cccdee6aa8bae46bcfbf6448518ebb6011660e5680f59fc70df\": container with ID starting with e1e487e2f6eb6cccdee6aa8bae46bcfbf6448518ebb6011660e5680f59fc70df not found: ID does not exist" containerID="e1e487e2f6eb6cccdee6aa8bae46bcfbf6448518ebb6011660e5680f59fc70df" Oct 30 19:27:01.811496 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.811461 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e487e2f6eb6cccdee6aa8bae46bcfbf6448518ebb6011660e5680f59fc70df"} err="failed to get container status \"e1e487e2f6eb6cccdee6aa8bae46bcfbf6448518ebb6011660e5680f59fc70df\": rpc error: code = NotFound desc = could not find container \"e1e487e2f6eb6cccdee6aa8bae46bcfbf6448518ebb6011660e5680f59fc70df\": container with ID starting with e1e487e2f6eb6cccdee6aa8bae46bcfbf6448518ebb6011660e5680f59fc70df not found: ID does not exist" Oct 30 19:27:01.811496 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.811479 4433 scope.go:122] "RemoveContainer" containerID="500575ebbf34c7b7126618dcc0a1214617a9a693f6f466be551cd1a208911d4a" Oct 30 19:27:01.818632 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.818617 4433 scope.go:122] "RemoveContainer" containerID="500575ebbf34c7b7126618dcc0a1214617a9a693f6f466be551cd1a208911d4a" Oct 30 19:27:01.818882 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:27:01.818865 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"500575ebbf34c7b7126618dcc0a1214617a9a693f6f466be551cd1a208911d4a\": container with ID starting with 500575ebbf34c7b7126618dcc0a1214617a9a693f6f466be551cd1a208911d4a not found: ID does not exist" containerID="500575ebbf34c7b7126618dcc0a1214617a9a693f6f466be551cd1a208911d4a" Oct 30 19:27:01.818927 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.818891 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"500575ebbf34c7b7126618dcc0a1214617a9a693f6f466be551cd1a208911d4a"} err="failed to get container status \"500575ebbf34c7b7126618dcc0a1214617a9a693f6f466be551cd1a208911d4a\": rpc error: code = NotFound desc = could not find container \"500575ebbf34c7b7126618dcc0a1214617a9a693f6f466be551cd1a208911d4a\": container with ID starting with 500575ebbf34c7b7126618dcc0a1214617a9a693f6f466be551cd1a208911d4a not found: ID does not exist" Oct 30 19:27:01.963606 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.963568 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-str7s"] Oct 30 19:27:01.966973 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:01.966944 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-str7s"] Oct 30 19:27:02.082296 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:02.082261 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57d457b6df-hg7m5"] Oct 30 19:27:02.088089 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:02.088063 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57d457b6df-hg7m5"] Oct 30 19:27:03.211130 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:03.211091 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7768d06b-2aaf-4226-b162-3228bf2a7459" path="/var/lib/kubelet/pods/7768d06b-2aaf-4226-b162-3228bf2a7459/volumes" Oct 30 19:27:03.211738 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:03.211540 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8df6052a-1078-4c61-826f-7ce6d7ac0811" path="/var/lib/kubelet/pods/8df6052a-1078-4c61-826f-7ce6d7ac0811/volumes" Oct 30 19:27:03.211914 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:03.211895 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b20c1fcc-acca-448e-a906-63bcf1b9a07f" path="/var/lib/kubelet/pods/b20c1fcc-acca-448e-a906-63bcf1b9a07f/volumes" Oct 30 19:27:06.286123 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:06.286031 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:18.495378 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:18.495342 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:18.728685 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:18.728647 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:18.963267 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:18.963235 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:23.046757 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:23.046722 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:23.285776 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:23.285741 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:23.365947 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:27:23.365910 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:27:27.781277 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:27.781241 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:28.378377 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:28.378346 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:51.383291 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:51.383254 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:52.670705 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:52.670668 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:52.985483 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:52.985401 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:53.365801 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:27:53.365762 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:27:54.609994 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:54.609934 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:54.839271 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:54.839242 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:55.068210 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.068164 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:55.577152 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.577121 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2"] Oct 30 19:27:55.577435 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.577424 4433 cpu_manager.go:414] "containerMap: removing container" podUID="7768d06b-2aaf-4226-b162-3228bf2a7459" containerName="authorino" Oct 30 19:27:55.577478 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.577436 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7768d06b-2aaf-4226-b162-3228bf2a7459" containerName="authorino" Oct 30 19:27:55.577478 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.577447 4433 cpu_manager.go:414] "containerMap: removing container" podUID="8df6052a-1078-4c61-826f-7ce6d7ac0811" containerName="authorino" Oct 30 19:27:55.577478 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.577452 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df6052a-1078-4c61-826f-7ce6d7ac0811" containerName="authorino" Oct 30 19:27:55.577567 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.577480 4433 cpu_manager.go:414] "containerMap: removing container" podUID="b20c1fcc-acca-448e-a906-63bcf1b9a07f" containerName="authorino" Oct 30 19:27:55.577567 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.577485 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b20c1fcc-acca-448e-a906-63bcf1b9a07f" containerName="authorino" Oct 30 19:27:55.577567 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.577533 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b20c1fcc-acca-448e-a906-63bcf1b9a07f" containerName="authorino" Oct 30 19:27:55.577567 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.577541 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="7768d06b-2aaf-4226-b162-3228bf2a7459" containerName="authorino" Oct 30 19:27:55.577567 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.577547 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df6052a-1078-4c61-826f-7ce6d7ac0811" containerName="authorino" Oct 30 19:27:55.581939 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.581922 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:27:55.686181 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.686147 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 19:27:55.686181 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.686155 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8c9cv\"" Oct 30 19:27:55.686653 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.686360 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:27:55.686653 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.686490 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:27:55.687471 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.687450 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2"] Oct 30 19:27:55.693568 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.693551 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcn99\" (UniqueName: \"kubernetes.io/projected/860c3b62-aa0e-466b-9991-029a5cf5a394-kube-api-access-hcn99\") pod \"maas-api-5c48877c75-hz2x2\" (UID: \"860c3b62-aa0e-466b-9991-029a5cf5a394\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:27:55.693629 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.693615 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/860c3b62-aa0e-466b-9991-029a5cf5a394-metrics-certs\") pod \"maas-api-5c48877c75-hz2x2\" (UID: \"860c3b62-aa0e-466b-9991-029a5cf5a394\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:27:55.693695 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.693678 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860c3b62-aa0e-466b-9991-029a5cf5a394-openshift-service-ca\") pod \"maas-api-5c48877c75-hz2x2\" (UID: \"860c3b62-aa0e-466b-9991-029a5cf5a394\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:27:55.693743 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.693734 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860c3b62-aa0e-466b-9991-029a5cf5a394-maas-api-tls\") pod \"maas-api-5c48877c75-hz2x2\" (UID: \"860c3b62-aa0e-466b-9991-029a5cf5a394\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:27:55.735598 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.735572 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:27:55.795148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.795110 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/860c3b62-aa0e-466b-9991-029a5cf5a394-metrics-certs\") pod \"maas-api-5c48877c75-hz2x2\" (UID: \"860c3b62-aa0e-466b-9991-029a5cf5a394\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:27:55.795148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.795151 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860c3b62-aa0e-466b-9991-029a5cf5a394-openshift-service-ca\") pod \"maas-api-5c48877c75-hz2x2\" (UID: \"860c3b62-aa0e-466b-9991-029a5cf5a394\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:27:55.795385 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.795189 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860c3b62-aa0e-466b-9991-029a5cf5a394-maas-api-tls\") pod \"maas-api-5c48877c75-hz2x2\" (UID: \"860c3b62-aa0e-466b-9991-029a5cf5a394\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:27:55.795385 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.795226 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcn99\" (UniqueName: \"kubernetes.io/projected/860c3b62-aa0e-466b-9991-029a5cf5a394-kube-api-access-hcn99\") pod \"maas-api-5c48877c75-hz2x2\" (UID: \"860c3b62-aa0e-466b-9991-029a5cf5a394\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:27:55.795851 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.795825 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860c3b62-aa0e-466b-9991-029a5cf5a394-openshift-service-ca\") pod \"maas-api-5c48877c75-hz2x2\" (UID: \"860c3b62-aa0e-466b-9991-029a5cf5a394\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:27:55.797560 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.797529 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860c3b62-aa0e-466b-9991-029a5cf5a394-maas-api-tls\") pod \"maas-api-5c48877c75-hz2x2\" (UID: \"860c3b62-aa0e-466b-9991-029a5cf5a394\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:27:55.797651 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.797562 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/860c3b62-aa0e-466b-9991-029a5cf5a394-metrics-certs\") pod \"maas-api-5c48877c75-hz2x2\" (UID: \"860c3b62-aa0e-466b-9991-029a5cf5a394\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:27:55.851504 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.851432 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcn99\" (UniqueName: \"kubernetes.io/projected/860c3b62-aa0e-466b-9991-029a5cf5a394-kube-api-access-hcn99\") pod \"maas-api-5c48877c75-hz2x2\" (UID: \"860c3b62-aa0e-466b-9991-029a5cf5a394\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:27:55.894485 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:55.894447 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:27:56.269063 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:56.268838 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2"] Oct 30 19:27:56.271560 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:27:56.271535 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860c3b62_aa0e_466b_9991_029a5cf5a394.slice/crio-b7c14cac48a8db2330d6df265c3304a4893a6c2b131c401947f26e221b26fef7 WatchSource:0}: Error finding container b7c14cac48a8db2330d6df265c3304a4893a6c2b131c401947f26e221b26fef7: Status 404 returned error can't find the container with id b7c14cac48a8db2330d6df265c3304a4893a6c2b131c401947f26e221b26fef7 Oct 30 19:27:56.624833 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:56.624802 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:57.039153 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:27:57.039065 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" event={"ID":"860c3b62-aa0e-466b-9991-029a5cf5a394","Type":"ContainerStarted","Data":"b7c14cac48a8db2330d6df265c3304a4893a6c2b131c401947f26e221b26fef7"} Oct 30 19:28:00.055003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:28:00.054945 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" event={"ID":"860c3b62-aa0e-466b-9991-029a5cf5a394","Type":"ContainerStarted","Data":"e46b255da168032a2f553a172d19f7ed857ec4f4c39351e9c85e2c01a2bb99aa"} Oct 30 19:28:00.055419 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:28:00.055056 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:28:00.170603 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:28:00.170545 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" podStartSLOduration=2.085622923 podStartE2EDuration="5.170530598s" totalImagesPullingTime="3.084907675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:00.16933325 +0000 UTC m=+727.104120775" watchObservedRunningTime="2026-10-30 19:28:00.170530598 +0000 UTC m=+727.105318122" Oct 30 19:28:06.064117 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:28:06.064063 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" podUID="860c3b62-aa0e-466b-9991-029a5cf5a394" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.35:8443/health\": dial tcp 10.134.0.35:8443: connect: connection refused" Oct 30 19:28:06.638345 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:28:06.638313 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:10.099051 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:28:10.099019 4433 generic.go:356] "Generic (PLEG): container finished" podID="860c3b62-aa0e-466b-9991-029a5cf5a394" containerID="e46b255da168032a2f553a172d19f7ed857ec4f4c39351e9c85e2c01a2bb99aa" exitCode=1 Oct 30 19:28:10.099474 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:28:10.099093 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" event={"ID":"860c3b62-aa0e-466b-9991-029a5cf5a394","Type":"ContainerDied","Data":"e46b255da168032a2f553a172d19f7ed857ec4f4c39351e9c85e2c01a2bb99aa"} Oct 30 19:28:10.099474 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:28:10.099408 4433 scope.go:122] "RemoveContainer" containerID="e46b255da168032a2f553a172d19f7ed857ec4f4c39351e9c85e2c01a2bb99aa" Oct 30 19:28:11.105882 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:28:11.105841 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" event={"ID":"860c3b62-aa0e-466b-9991-029a5cf5a394","Type":"ContainerStarted","Data":"1ac50935d0e20a623605665b66d5a2dfd64340b8d0c35507268cc794b7b5057c"} Oct 30 19:28:11.106290 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:28:11.106099 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:28:17.115809 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:28:17.115779 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 19:28:23.365674 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:28:23.365636 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:28:51.706190 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:28:51.706143 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:52.965903 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:28:52.965867 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:53.365059 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:28:53.365021 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:28:53.923702 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:28:53.923656 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:02.156678 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:02.156642 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:02.399515 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:02.399478 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:09.241847 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:09.241814 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:21.370642 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:21.370611 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:23.365120 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:29:23.365080 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:29:25.772739 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:25.772697 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs"] Oct 30 19:29:25.776186 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:25.776160 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:25.828200 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:25.828166 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 19:29:25.877346 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:25.877306 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 19:29:25.877346 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:25.877313 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2svj8\"" Oct 30 19:29:25.877574 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:25.877421 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 19:29:25.878922 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:25.878900 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs"] Oct 30 19:29:25.943085 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:25.943057 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da30980c-5b69-448d-a0c3-b2e21f677d8f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:25.943268 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:25.943125 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbbw7\" (UniqueName: \"kubernetes.io/projected/da30980c-5b69-448d-a0c3-b2e21f677d8f-kube-api-access-gbbw7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:25.943268 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:25.943198 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da30980c-5b69-448d-a0c3-b2e21f677d8f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:25.943268 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:25.943244 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da30980c-5b69-448d-a0c3-b2e21f677d8f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:25.943386 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:25.943317 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da30980c-5b69-448d-a0c3-b2e21f677d8f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:25.943386 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:25.943358 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da30980c-5b69-448d-a0c3-b2e21f677d8f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:25.954305 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:25.954254 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:26.044363 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:26.044292 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da30980c-5b69-448d-a0c3-b2e21f677d8f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:26.044363 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:26.044332 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbbw7\" (UniqueName: \"kubernetes.io/projected/da30980c-5b69-448d-a0c3-b2e21f677d8f-kube-api-access-gbbw7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:26.044363 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:26.044349 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da30980c-5b69-448d-a0c3-b2e21f677d8f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:26.044545 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:26.044378 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da30980c-5b69-448d-a0c3-b2e21f677d8f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:26.044545 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:26.044400 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da30980c-5b69-448d-a0c3-b2e21f677d8f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:26.044545 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:26.044428 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da30980c-5b69-448d-a0c3-b2e21f677d8f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:26.044841 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:26.044820 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da30980c-5b69-448d-a0c3-b2e21f677d8f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:26.044892 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:26.044837 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da30980c-5b69-448d-a0c3-b2e21f677d8f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:26.044892 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:26.044882 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da30980c-5b69-448d-a0c3-b2e21f677d8f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:26.046724 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:26.046706 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da30980c-5b69-448d-a0c3-b2e21f677d8f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:26.047006 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:26.046973 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da30980c-5b69-448d-a0c3-b2e21f677d8f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:26.101636 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:26.101609 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbbw7\" (UniqueName: \"kubernetes.io/projected/da30980c-5b69-448d-a0c3-b2e21f677d8f-kube-api-access-gbbw7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f6zfs\" (UID: \"da30980c-5b69-448d-a0c3-b2e21f677d8f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:26.388706 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:26.388667 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:26.759086 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:26.759060 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs"] Oct 30 19:29:26.761463 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:29:26.761438 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda30980c_5b69_448d_a0c3_b2e21f677d8f.slice/crio-be53b3f79d1ccb05377e9bcbf570294375d85b920a37f22d18c90b089bf53080 WatchSource:0}: Error finding container be53b3f79d1ccb05377e9bcbf570294375d85b920a37f22d18c90b089bf53080: Status 404 returned error can't find the container with id be53b3f79d1ccb05377e9bcbf570294375d85b920a37f22d18c90b089bf53080 Oct 30 19:29:27.411440 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:27.411406 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" event={"ID":"da30980c-5b69-448d-a0c3-b2e21f677d8f","Type":"ContainerStarted","Data":"be53b3f79d1ccb05377e9bcbf570294375d85b920a37f22d18c90b089bf53080"} Oct 30 19:29:29.435291 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:29.435249 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:29.439820 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:29.439779 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" event={"ID":"da30980c-5b69-448d-a0c3-b2e21f677d8f","Type":"ContainerStarted","Data":"fdb870ce72f1b53625782a3ee716b651d520a2a8b572f785197d9936d5b2f878"} Oct 30 19:29:30.389095 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:30.389060 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:32.387709 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.387656 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" podStartSLOduration=5.73795254 podStartE2EDuration="7.387638944s" totalImagesPullingTime="1.649686404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:29.569055016 +0000 UTC m=+816.503842542" watchObservedRunningTime="2026-10-30 19:29:32.387638944 +0000 UTC m=+819.322426465" Oct 30 19:29:32.388143 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.388067 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r"] Oct 30 19:29:32.391530 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.391513 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.393652 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.393628 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51dce2ca-7adf-4e00-8d76-c32a82932883-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.393740 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.393666 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51dce2ca-7adf-4e00-8d76-c32a82932883-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.393740 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.393699 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51dce2ca-7adf-4e00-8d76-c32a82932883-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.393825 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.393806 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51dce2ca-7adf-4e00-8d76-c32a82932883-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.393861 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.393847 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4rjl\" (UniqueName: \"kubernetes.io/projected/51dce2ca-7adf-4e00-8d76-c32a82932883-kube-api-access-f4rjl\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.393890 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.393877 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51dce2ca-7adf-4e00-8d76-c32a82932883-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.460678 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.460649 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 19:29:32.494460 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.494419 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51dce2ca-7adf-4e00-8d76-c32a82932883-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.494647 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.494469 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51dce2ca-7adf-4e00-8d76-c32a82932883-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.494647 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.494493 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4rjl\" (UniqueName: \"kubernetes.io/projected/51dce2ca-7adf-4e00-8d76-c32a82932883-kube-api-access-f4rjl\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.494647 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.494514 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51dce2ca-7adf-4e00-8d76-c32a82932883-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.494647 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.494539 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51dce2ca-7adf-4e00-8d76-c32a82932883-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.494647 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.494465 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r"] Oct 30 19:29:32.494647 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.494570 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51dce2ca-7adf-4e00-8d76-c32a82932883-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.494874 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.494862 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51dce2ca-7adf-4e00-8d76-c32a82932883-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.494955 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.494933 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51dce2ca-7adf-4e00-8d76-c32a82932883-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.495016 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.494958 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51dce2ca-7adf-4e00-8d76-c32a82932883-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.496763 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.496738 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51dce2ca-7adf-4e00-8d76-c32a82932883-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.497180 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.497157 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51dce2ca-7adf-4e00-8d76-c32a82932883-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.551235 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.551204 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4rjl\" (UniqueName: \"kubernetes.io/projected/51dce2ca-7adf-4e00-8d76-c32a82932883-kube-api-access-f4rjl\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x652r\" (UID: \"51dce2ca-7adf-4e00-8d76-c32a82932883\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:32.575094 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.575061 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:32.704613 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:32.704517 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:33.074828 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:33.074802 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r"] Oct 30 19:29:33.077097 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:29:33.077072 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51dce2ca_7adf_4e00_8d76_c32a82932883.slice/crio-764211060d657666bc6442995d3099ca45018b4e6f4b3be9eb361abbd41913de WatchSource:0}: Error finding container 764211060d657666bc6442995d3099ca45018b4e6f4b3be9eb361abbd41913de: Status 404 returned error can't find the container with id 764211060d657666bc6442995d3099ca45018b4e6f4b3be9eb361abbd41913de Oct 30 19:29:33.458197 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:33.458114 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" event={"ID":"51dce2ca-7adf-4e00-8d76-c32a82932883","Type":"ContainerStarted","Data":"764211060d657666bc6442995d3099ca45018b4e6f4b3be9eb361abbd41913de"} Oct 30 19:29:34.464107 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:34.464056 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" event={"ID":"51dce2ca-7adf-4e00-8d76-c32a82932883","Type":"ContainerStarted","Data":"248ad878f4c643febb4a4aca2e537129ba8613dbd48da833109a64073cdeba6f"} Oct 30 19:29:34.602060 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:34.601967 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" podStartSLOduration=1.913279444 podStartE2EDuration="2.601953178s" totalImagesPullingTime="688.673734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:34.601716348 +0000 UTC m=+821.536503886" watchObservedRunningTime="2026-10-30 19:29:34.601953178 +0000 UTC m=+821.536740702" Oct 30 19:29:34.705112 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:34.705073 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:35.975921 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:35.975886 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:36.389147 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:36.389111 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:36.401896 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:36.401867 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:36.484715 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:36.484686 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f6zfs" Oct 30 19:29:42.484844 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.484805 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk"] Oct 30 19:29:42.488273 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.488252 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.574352 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.574314 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.574352 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.574353 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwxkn\" (UniqueName: \"kubernetes.io/projected/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-kube-api-access-pwxkn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.574554 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.574377 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.574554 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.574502 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.574554 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.574543 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.574647 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.574615 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.589265 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.589241 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 19:29:42.590713 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.590695 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk"] Oct 30 19:29:42.675069 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.675038 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.675254 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.675112 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.675254 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.675140 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.675254 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.675171 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.675254 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.675187 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:42.675254 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.675204 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.675254 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.675237 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwxkn\" (UniqueName: \"kubernetes.io/projected/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-kube-api-access-pwxkn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.675598 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.675573 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.675706 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.675610 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.675754 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.675707 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.677441 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.677417 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.677584 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.677568 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.705069 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.705051 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:42.717125 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.717106 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:42.740119 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.740064 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwxkn\" (UniqueName: \"kubernetes.io/projected/ee9f5fbf-a71a-491e-abd3-8b59a15066ae-kube-api-access-pwxkn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk\" (UID: \"ee9f5fbf-a71a-491e-abd3-8b59a15066ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:42.799839 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:42.799815 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:43.167866 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:43.167834 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk"] Oct 30 19:29:43.170957 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:29:43.170930 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee9f5fbf_a71a_491e_abd3_8b59a15066ae.slice/crio-220ac7acc67ffe07ba6a92a645ff4d6d48393ead3d7af0af32cbd9c80ac9ed31 WatchSource:0}: Error finding container 220ac7acc67ffe07ba6a92a645ff4d6d48393ead3d7af0af32cbd9c80ac9ed31: Status 404 returned error can't find the container with id 220ac7acc67ffe07ba6a92a645ff4d6d48393ead3d7af0af32cbd9c80ac9ed31 Oct 30 19:29:43.504236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:43.504150 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" event={"ID":"ee9f5fbf-a71a-491e-abd3-8b59a15066ae","Type":"ContainerStarted","Data":"220ac7acc67ffe07ba6a92a645ff4d6d48393ead3d7af0af32cbd9c80ac9ed31"} Oct 30 19:29:43.514968 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:43.514943 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x652r" Oct 30 19:29:44.517915 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:44.517874 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" event={"ID":"ee9f5fbf-a71a-491e-abd3-8b59a15066ae","Type":"ContainerStarted","Data":"aea875a05bb7d0b917a3f1aad3a71785f226ef9065eaea94f1a4dd2a2f49d2ce"} Oct 30 19:29:44.632537 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:44.632485 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" podStartSLOduration=1.91724494 podStartE2EDuration="2.632472618s" totalImagesPullingTime="715.227678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:44.631358777 +0000 UTC m=+831.566146298" watchObservedRunningTime="2026-10-30 19:29:44.632472618 +0000 UTC m=+831.567260142" Oct 30 19:29:44.800756 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:44.800681 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:46.514083 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:46.514030 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:52.051897 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:52.051859 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:52.800892 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:52.800856 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:52.813068 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:52.813043 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:53.364345 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:29:53.364304 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:29:53.568898 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:53.568870 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:29:55.854177 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.854137 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c"] Oct 30 19:29:55.857659 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.857637 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.882930 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.882905 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.883075 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.882938 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.883075 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.882955 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.883075 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.883038 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.883173 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.883078 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwmq7\" (UniqueName: \"kubernetes.io/projected/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-kube-api-access-mwmq7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.883173 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.883105 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.883235 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.883172 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.883235 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.883213 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.883294 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.883234 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.903544 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.903516 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4"] Oct 30 19:29:55.907077 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.907059 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:55.918910 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.918893 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f2npg\"" Oct 30 19:29:55.962534 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.962508 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c"] Oct 30 19:29:55.984544 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.984506 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.984759 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.984557 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:55.984759 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.984574 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:55.984759 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.984595 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwmq7\" (UniqueName: \"kubernetes.io/projected/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-kube-api-access-mwmq7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.984923 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.984762 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.984923 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.984882 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:55.985084 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.984925 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.985084 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.984949 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.985084 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.985013 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.985084 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.985047 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8jbf\" (UniqueName: \"kubernetes.io/projected/758501e2-8b8f-4465-93d3-6ba4ef2a033a-kube-api-access-v8jbf\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:55.985084 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.985079 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:55.985328 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.985139 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:55.985328 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.985164 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:55.985328 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.985317 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:55.985465 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.985337 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.985465 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.985355 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.985465 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.985045 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.985611 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.985534 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.985611 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.985563 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:55.985611 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.985593 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.985760 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.985665 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.985817 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.985793 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.985923 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.985905 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.987361 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.987333 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:55.987548 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:55.987530 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:56.013032 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.013002 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4"] Oct 30 19:29:56.043110 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.043077 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:56.043110 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.043103 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwmq7\" (UniqueName: \"kubernetes.io/projected/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-kube-api-access-mwmq7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5n86c\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:56.086632 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.086603 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8jbf\" (UniqueName: \"kubernetes.io/projected/758501e2-8b8f-4465-93d3-6ba4ef2a033a-kube-api-access-v8jbf\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.086792 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.086644 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.086792 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.086688 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.086792 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.086711 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.086792 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.086764 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.087012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.086795 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.087012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.086846 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.087012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.086872 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.087012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.086910 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.087170 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.087143 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.087490 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.087459 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.087600 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.087465 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.087600 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.087531 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.087600 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.087556 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.089095 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.089073 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.089252 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.089237 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.144595 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.144536 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.144704 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.144674 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8jbf\" (UniqueName: \"kubernetes.io/projected/758501e2-8b8f-4465-93d3-6ba4ef2a033a-kube-api-access-v8jbf\") pod \"maas-default-gateway-openshift-default-6db586dd6c-9wgm4\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.170549 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.170515 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:56.219403 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.219370 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:56.599271 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.599242 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c"] Oct 30 19:29:56.599550 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:29:56.599527 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0046a0_bdf5_4975_a4e1_c40e43b7a7ac.slice/crio-21496e222dfa1f4342d4a2c01e983a1627012d3aa41cbf3a321f29f143360540 WatchSource:0}: Error finding container 21496e222dfa1f4342d4a2c01e983a1627012d3aa41cbf3a321f29f143360540: Status 404 returned error can't find the container with id 21496e222dfa1f4342d4a2c01e983a1627012d3aa41cbf3a321f29f143360540 Oct 30 19:29:56.601523 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.601493 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:56.601603 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.601557 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:56.601603 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.601588 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:56.606103 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.606083 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4"] Oct 30 19:29:56.649525 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:29:56.649500 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758501e2_8b8f_4465_93d3_6ba4ef2a033a.slice/crio-545bf861220f0075e2f856cf3f581b28a2e0d335a65c03a8dfc63e56c90008ff WatchSource:0}: Error finding container 545bf861220f0075e2f856cf3f581b28a2e0d335a65c03a8dfc63e56c90008ff: Status 404 returned error can't find the container with id 545bf861220f0075e2f856cf3f581b28a2e0d335a65c03a8dfc63e56c90008ff Oct 30 19:29:56.651768 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.651739 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:56.651878 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.651804 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:56.651878 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:56.651834 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:57.578892 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:57.578856 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" event={"ID":"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac","Type":"ContainerStarted","Data":"3aecb66cd98f421e2448b3a39de52c44c9b98712bb2c7eb5eb8d17de972faa5f"} Oct 30 19:29:57.579294 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:57.578921 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" event={"ID":"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac","Type":"ContainerStarted","Data":"21496e222dfa1f4342d4a2c01e983a1627012d3aa41cbf3a321f29f143360540"} Oct 30 19:29:57.580365 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:57.580337 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" event={"ID":"758501e2-8b8f-4465-93d3-6ba4ef2a033a","Type":"ContainerStarted","Data":"0e1222e4bf323bcf1651fb44038472726c1ff50c286c208bd360e1e89fe317d3"} Oct 30 19:29:57.580497 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:57.580382 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" event={"ID":"758501e2-8b8f-4465-93d3-6ba4ef2a033a","Type":"ContainerStarted","Data":"545bf861220f0075e2f856cf3f581b28a2e0d335a65c03a8dfc63e56c90008ff"} Oct 30 19:29:57.828360 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:57.828296 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" podStartSLOduration=2.828274847 podStartE2EDuration="2.828274847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:57.70843341 +0000 UTC m=+844.643220947" watchObservedRunningTime="2026-10-30 19:29:57.828274847 +0000 UTC m=+844.763062372" Oct 30 19:29:57.835079 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:57.834624 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" podStartSLOduration=2.834605495 podStartE2EDuration="2.834605495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:57.834132905 +0000 UTC m=+844.768920420" watchObservedRunningTime="2026-10-30 19:29:57.834605495 +0000 UTC m=+844.769393023" Oct 30 19:29:58.171696 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:58.171608 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:29:58.172851 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:58.172828 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5n86c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:29:58.172916 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:58.172890 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" podUID="9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 19:29:58.228307 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:58.226773 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:29:58.228616 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:58.228580 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-9wgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:29:58.228784 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:58.228758 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" podUID="758501e2-8b8f-4465-93d3-6ba4ef2a033a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:29:59.171866 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:59.171827 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5n86c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:29:59.172282 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:59.171893 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" podUID="9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 19:29:59.224002 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:59.222303 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-9wgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:29:59.224002 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:29:59.222373 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" podUID="758501e2-8b8f-4465-93d3-6ba4ef2a033a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:30:00.170899 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:00.170854 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5n86c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:30:00.171072 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:00.170913 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" podUID="9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 19:30:00.222488 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:00.222362 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-9wgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:30:00.222488 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:00.222424 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" podUID="758501e2-8b8f-4465-93d3-6ba4ef2a033a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:30:01.171856 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:01.171822 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5n86c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:30:01.172059 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:01.171891 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" podUID="9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 19:30:01.229022 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:01.228334 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-9wgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:30:01.229022 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:01.228436 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" podUID="758501e2-8b8f-4465-93d3-6ba4ef2a033a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:30:02.170841 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:02.170802 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5n86c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:30:02.171048 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:02.170860 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" podUID="9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 19:30:02.226866 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:02.226826 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-9wgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:30:02.227244 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:02.227212 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" podUID="758501e2-8b8f-4465-93d3-6ba4ef2a033a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:30:03.171724 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:03.171676 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5n86c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:30:03.172247 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:03.171747 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" podUID="9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 19:30:03.220021 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:03.219966 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-9wgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:30:03.220187 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:03.220047 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" podUID="758501e2-8b8f-4465-93d3-6ba4ef2a033a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:30:04.720457 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:04.720424 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:30:04.720920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:04.720824 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:30:04.722400 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:04.722377 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:30:04.731801 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:04.731781 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:30:04.732243 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:04.732192 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:30:04.733095 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:04.733077 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:30:19.685925 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:19.685884 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:20.523307 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:20.523277 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:22.001295 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:22.001262 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:23.365117 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:30:23.365079 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:30:27.299143 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:27.299111 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cb7567d84-s4vn7"] Oct 30 19:30:27.302488 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:27.302471 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cb7567d84-s4vn7" Oct 30 19:30:27.358747 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:27.358721 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cb7567d84-s4vn7"] Oct 30 19:30:27.398991 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:27.398947 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbht6\" (UniqueName: \"kubernetes.io/projected/f9b94090-336f-434e-a889-a9cb0b8f571c-kube-api-access-lbht6\") pod \"authorino-7cb7567d84-s4vn7\" (UID: \"f9b94090-336f-434e-a889-a9cb0b8f571c\") " pod="kuadrant-system/authorino-7cb7567d84-s4vn7" Oct 30 19:30:27.399151 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:27.399131 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f9b94090-336f-434e-a889-a9cb0b8f571c-tls-cert\") pod \"authorino-7cb7567d84-s4vn7\" (UID: \"f9b94090-336f-434e-a889-a9cb0b8f571c\") " pod="kuadrant-system/authorino-7cb7567d84-s4vn7" Oct 30 19:30:27.495721 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:27.495695 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:27.499922 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:27.499890 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f9b94090-336f-434e-a889-a9cb0b8f571c-tls-cert\") pod \"authorino-7cb7567d84-s4vn7\" (UID: \"f9b94090-336f-434e-a889-a9cb0b8f571c\") " pod="kuadrant-system/authorino-7cb7567d84-s4vn7" Oct 30 19:30:27.500050 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:27.499938 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbht6\" (UniqueName: \"kubernetes.io/projected/f9b94090-336f-434e-a889-a9cb0b8f571c-kube-api-access-lbht6\") pod \"authorino-7cb7567d84-s4vn7\" (UID: \"f9b94090-336f-434e-a889-a9cb0b8f571c\") " pod="kuadrant-system/authorino-7cb7567d84-s4vn7" Oct 30 19:30:27.502303 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:27.502282 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f9b94090-336f-434e-a889-a9cb0b8f571c-tls-cert\") pod \"authorino-7cb7567d84-s4vn7\" (UID: \"f9b94090-336f-434e-a889-a9cb0b8f571c\") " pod="kuadrant-system/authorino-7cb7567d84-s4vn7" Oct 30 19:30:27.556967 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:27.556898 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbht6\" (UniqueName: \"kubernetes.io/projected/f9b94090-336f-434e-a889-a9cb0b8f571c-kube-api-access-lbht6\") pod \"authorino-7cb7567d84-s4vn7\" (UID: \"f9b94090-336f-434e-a889-a9cb0b8f571c\") " pod="kuadrant-system/authorino-7cb7567d84-s4vn7" Oct 30 19:30:27.613319 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:27.613287 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cb7567d84-s4vn7" Oct 30 19:30:27.749996 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:27.749942 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:27.980997 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:27.980960 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cb7567d84-s4vn7"] Oct 30 19:30:27.983630 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:30:27.983603 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9b94090_336f_434e_a889_a9cb0b8f571c.slice/crio-ea6cda8786a3469248d6ad43b9f379b567ff19cc0c2ad701753f175d3895e323 WatchSource:0}: Error finding container ea6cda8786a3469248d6ad43b9f379b567ff19cc0c2ad701753f175d3895e323: Status 404 returned error can't find the container with id ea6cda8786a3469248d6ad43b9f379b567ff19cc0c2ad701753f175d3895e323 Oct 30 19:30:28.757347 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:28.757310 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cb7567d84-s4vn7" event={"ID":"f9b94090-336f-434e-a889-a9cb0b8f571c","Type":"ContainerStarted","Data":"ea6cda8786a3469248d6ad43b9f379b567ff19cc0c2ad701753f175d3895e323"} Oct 30 19:30:29.763788 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:29.763683 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cb7567d84-s4vn7" event={"ID":"f9b94090-336f-434e-a889-a9cb0b8f571c","Type":"ContainerStarted","Data":"9a187901f5b11cc765048012fc6f63fa64d558a051f8e0e3ffd532da9cbcb8d5"} Oct 30 19:30:29.878389 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:29.878337 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7cb7567d84-s4vn7" podStartSLOduration=1.407062274 podStartE2EDuration="2.878324505s" totalImagesPullingTime="1.471262231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:29.876432516 +0000 UTC m=+876.811220040" watchObservedRunningTime="2026-10-30 19:30:29.878324505 +0000 UTC m=+876.813112028" Oct 30 19:30:29.927392 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:29.927357 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f8947ddb7-8kgx9"] Oct 30 19:30:29.927597 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:29.927561 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f8947ddb7-8kgx9" podUID="2601bf00-9912-4a32-b570-6d5d1a47bec4" containerName="authorino" containerID="cri-o://6a25d1ea34d7d60c0a68f5d5a453f1f123d368cccc8e34219c6c56bf7093cb23" gracePeriod=30 Oct 30 19:30:30.172338 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:30.172314 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f8947ddb7-8kgx9" Oct 30 19:30:30.327646 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:30.327613 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2601bf00-9912-4a32-b570-6d5d1a47bec4-kube-api-access-klc6g\" (UniqueName: \"kubernetes.io/projected/2601bf00-9912-4a32-b570-6d5d1a47bec4-kube-api-access-klc6g\") pod \"2601bf00-9912-4a32-b570-6d5d1a47bec4\" (UID: \"2601bf00-9912-4a32-b570-6d5d1a47bec4\") " Oct 30 19:30:30.327646 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:30.327647 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2601bf00-9912-4a32-b570-6d5d1a47bec4-tls-cert\" (UniqueName: \"kubernetes.io/secret/2601bf00-9912-4a32-b570-6d5d1a47bec4-tls-cert\") pod \"2601bf00-9912-4a32-b570-6d5d1a47bec4\" (UID: \"2601bf00-9912-4a32-b570-6d5d1a47bec4\") " Oct 30 19:30:30.329685 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:30.329658 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2601bf00-9912-4a32-b570-6d5d1a47bec4-kube-api-access-klc6g" pod "2601bf00-9912-4a32-b570-6d5d1a47bec4" (UID: "2601bf00-9912-4a32-b570-6d5d1a47bec4"). InnerVolumeSpecName "kube-api-access-klc6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:30.338275 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:30.338251 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2601bf00-9912-4a32-b570-6d5d1a47bec4-tls-cert" pod "2601bf00-9912-4a32-b570-6d5d1a47bec4" (UID: "2601bf00-9912-4a32-b570-6d5d1a47bec4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:30.429336 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:30.429302 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klc6g\" (UniqueName: \"kubernetes.io/projected/2601bf00-9912-4a32-b570-6d5d1a47bec4-kube-api-access-klc6g\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:30.429336 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:30.429330 4433 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2601bf00-9912-4a32-b570-6d5d1a47bec4-tls-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:30.769893 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:30.769804 4433 generic.go:356] "Generic (PLEG): container finished" podID="2601bf00-9912-4a32-b570-6d5d1a47bec4" containerID="6a25d1ea34d7d60c0a68f5d5a453f1f123d368cccc8e34219c6c56bf7093cb23" exitCode=0 Oct 30 19:30:30.769893 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:30.769849 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f8947ddb7-8kgx9" Oct 30 19:30:30.770473 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:30.769886 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f8947ddb7-8kgx9" event={"ID":"2601bf00-9912-4a32-b570-6d5d1a47bec4","Type":"ContainerDied","Data":"6a25d1ea34d7d60c0a68f5d5a453f1f123d368cccc8e34219c6c56bf7093cb23"} Oct 30 19:30:30.770473 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:30.769936 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f8947ddb7-8kgx9" event={"ID":"2601bf00-9912-4a32-b570-6d5d1a47bec4","Type":"ContainerDied","Data":"383ee453bf6d559e6dbb9ef46dd33939f6667d78fb2737d6a33685053329ab6d"} Oct 30 19:30:30.770473 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:30.769958 4433 scope.go:122] "RemoveContainer" containerID="6a25d1ea34d7d60c0a68f5d5a453f1f123d368cccc8e34219c6c56bf7093cb23" Oct 30 19:30:30.779943 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:30.779914 4433 scope.go:122] "RemoveContainer" containerID="6a25d1ea34d7d60c0a68f5d5a453f1f123d368cccc8e34219c6c56bf7093cb23" Oct 30 19:30:30.780232 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:30:30.780212 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a25d1ea34d7d60c0a68f5d5a453f1f123d368cccc8e34219c6c56bf7093cb23\": container with ID starting with 6a25d1ea34d7d60c0a68f5d5a453f1f123d368cccc8e34219c6c56bf7093cb23 not found: ID does not exist" containerID="6a25d1ea34d7d60c0a68f5d5a453f1f123d368cccc8e34219c6c56bf7093cb23" Oct 30 19:30:30.780315 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:30.780241 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a25d1ea34d7d60c0a68f5d5a453f1f123d368cccc8e34219c6c56bf7093cb23"} err="failed to get container status \"6a25d1ea34d7d60c0a68f5d5a453f1f123d368cccc8e34219c6c56bf7093cb23\": rpc error: code = NotFound desc = could not find container \"6a25d1ea34d7d60c0a68f5d5a453f1f123d368cccc8e34219c6c56bf7093cb23\": container with ID starting with 6a25d1ea34d7d60c0a68f5d5a453f1f123d368cccc8e34219c6c56bf7093cb23 not found: ID does not exist" Oct 30 19:30:30.935168 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:30.935132 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f8947ddb7-8kgx9"] Oct 30 19:30:30.944951 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:30.944924 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f8947ddb7-8kgx9"] Oct 30 19:30:31.212799 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:31.212764 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2601bf00-9912-4a32-b570-6d5d1a47bec4" path="/var/lib/kubelet/pods/2601bf00-9912-4a32-b570-6d5d1a47bec4/volumes" Oct 30 19:30:35.926622 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:35.926592 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:36.161796 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.161761 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:36.219683 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.219613 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-t2t65"] Oct 30 19:30:36.219973 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.219961 4433 cpu_manager.go:414] "containerMap: removing container" podUID="2601bf00-9912-4a32-b570-6d5d1a47bec4" containerName="authorino" Oct 30 19:30:36.220041 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.219974 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2601bf00-9912-4a32-b570-6d5d1a47bec4" containerName="authorino" Oct 30 19:30:36.220077 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.220066 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2601bf00-9912-4a32-b570-6d5d1a47bec4" containerName="authorino" Oct 30 19:30:36.224294 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.224279 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t2t65" Oct 30 19:30:36.276479 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.276452 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 19:30:36.326013 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.325970 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5vdf8\"" Oct 30 19:30:36.327578 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.327556 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-t2t65"] Oct 30 19:30:36.377446 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.377414 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-69g25"] Oct 30 19:30:36.381182 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.381147 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-69g25" Oct 30 19:30:36.389008 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.388958 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a5fdf4-c366-4af9-b351-15a6617824d7-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-t2t65\" (UID: \"f8a5fdf4-c366-4af9-b351-15a6617824d7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t2t65" Oct 30 19:30:36.389152 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.389134 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gwqb\" (UniqueName: \"kubernetes.io/projected/f8a5fdf4-c366-4af9-b351-15a6617824d7-kube-api-access-2gwqb\") pod \"payload-pre-processing-7b98bb678c-t2t65\" (UID: \"f8a5fdf4-c366-4af9-b351-15a6617824d7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t2t65" Oct 30 19:30:36.432093 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.432052 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-69g25"] Oct 30 19:30:36.490583 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.490502 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a5fdf4-c366-4af9-b351-15a6617824d7-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-t2t65\" (UID: \"f8a5fdf4-c366-4af9-b351-15a6617824d7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t2t65" Oct 30 19:30:36.490583 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.490547 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgr2f\" (UniqueName: \"kubernetes.io/projected/da86486a-f4f9-4bc3-a5e0-be4073682c0c-kube-api-access-rgr2f\") pod \"payload-processing-5f78687d5b-69g25\" (UID: \"da86486a-f4f9-4bc3-a5e0-be4073682c0c\") " pod="openshift-ingress/payload-processing-5f78687d5b-69g25" Oct 30 19:30:36.490779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.490595 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gwqb\" (UniqueName: \"kubernetes.io/projected/f8a5fdf4-c366-4af9-b351-15a6617824d7-kube-api-access-2gwqb\") pod \"payload-pre-processing-7b98bb678c-t2t65\" (UID: \"f8a5fdf4-c366-4af9-b351-15a6617824d7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t2t65" Oct 30 19:30:36.490779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.490616 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da86486a-f4f9-4bc3-a5e0-be4073682c0c-plugins-config-volume\") pod \"payload-processing-5f78687d5b-69g25\" (UID: \"da86486a-f4f9-4bc3-a5e0-be4073682c0c\") " pod="openshift-ingress/payload-processing-5f78687d5b-69g25" Oct 30 19:30:36.491521 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.491489 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a5fdf4-c366-4af9-b351-15a6617824d7-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-t2t65\" (UID: \"f8a5fdf4-c366-4af9-b351-15a6617824d7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t2t65" Oct 30 19:30:36.553378 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.553345 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gwqb\" (UniqueName: \"kubernetes.io/projected/f8a5fdf4-c366-4af9-b351-15a6617824d7-kube-api-access-2gwqb\") pod \"payload-pre-processing-7b98bb678c-t2t65\" (UID: \"f8a5fdf4-c366-4af9-b351-15a6617824d7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t2t65" Oct 30 19:30:36.591563 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.591527 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da86486a-f4f9-4bc3-a5e0-be4073682c0c-plugins-config-volume\") pod \"payload-processing-5f78687d5b-69g25\" (UID: \"da86486a-f4f9-4bc3-a5e0-be4073682c0c\") " pod="openshift-ingress/payload-processing-5f78687d5b-69g25" Oct 30 19:30:36.591738 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.591643 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgr2f\" (UniqueName: \"kubernetes.io/projected/da86486a-f4f9-4bc3-a5e0-be4073682c0c-kube-api-access-rgr2f\") pod \"payload-processing-5f78687d5b-69g25\" (UID: \"da86486a-f4f9-4bc3-a5e0-be4073682c0c\") " pod="openshift-ingress/payload-processing-5f78687d5b-69g25" Oct 30 19:30:36.592210 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.592187 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da86486a-f4f9-4bc3-a5e0-be4073682c0c-plugins-config-volume\") pod \"payload-processing-5f78687d5b-69g25\" (UID: \"da86486a-f4f9-4bc3-a5e0-be4073682c0c\") " pod="openshift-ingress/payload-processing-5f78687d5b-69g25" Oct 30 19:30:36.652989 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.652955 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgr2f\" (UniqueName: \"kubernetes.io/projected/da86486a-f4f9-4bc3-a5e0-be4073682c0c-kube-api-access-rgr2f\") pod \"payload-processing-5f78687d5b-69g25\" (UID: \"da86486a-f4f9-4bc3-a5e0-be4073682c0c\") " pod="openshift-ingress/payload-processing-5f78687d5b-69g25" Oct 30 19:30:36.696788 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.696751 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-69g25" Oct 30 19:30:36.837094 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:36.837057 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t2t65" Oct 30 19:30:37.068820 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:37.068796 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-69g25"] Oct 30 19:30:37.070781 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:30:37.070744 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda86486a_f4f9_4bc3_a5e0_be4073682c0c.slice/crio-ed3805383bba1df05eb60cf3d02019c5ab56667ecc38e50a2e14307ce2e272c2 WatchSource:0}: Error finding container ed3805383bba1df05eb60cf3d02019c5ab56667ecc38e50a2e14307ce2e272c2: Status 404 returned error can't find the container with id ed3805383bba1df05eb60cf3d02019c5ab56667ecc38e50a2e14307ce2e272c2 Oct 30 19:30:37.219507 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:37.219470 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-t2t65"] Oct 30 19:30:37.808001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:37.807862 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t2t65" event={"ID":"f8a5fdf4-c366-4af9-b351-15a6617824d7","Type":"ContainerStarted","Data":"30629cd2f4aa11caf7426ddeb899132a1e11f223ec077fcaf3627421bef112ef"} Oct 30 19:30:37.809654 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:37.809618 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-69g25" event={"ID":"da86486a-f4f9-4bc3-a5e0-be4073682c0c","Type":"ContainerStarted","Data":"ed3805383bba1df05eb60cf3d02019c5ab56667ecc38e50a2e14307ce2e272c2"} Oct 30 19:30:40.826043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:40.826008 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-69g25" event={"ID":"da86486a-f4f9-4bc3-a5e0-be4073682c0c","Type":"ContainerStarted","Data":"84addad7559cd99e9bb23f77286f9510f96b48236e76064a2c2bda89ee8873aa"} Oct 30 19:30:40.826528 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:40.826093 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-69g25" Oct 30 19:30:40.827527 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:40.827504 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t2t65" event={"ID":"f8a5fdf4-c366-4af9-b351-15a6617824d7","Type":"ContainerStarted","Data":"47ee833dbbd39c844661815e59c83cf91bed74142dcbae84890aae52bd66189b"} Oct 30 19:30:40.827635 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:40.827620 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t2t65" Oct 30 19:30:40.939331 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:40.939285 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-69g25" podStartSLOduration=1.909447017 podStartE2EDuration="4.939274248s" totalImagesPullingTime="3.029827231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:40.939080531 +0000 UTC m=+887.873868056" watchObservedRunningTime="2026-10-30 19:30:40.939274248 +0000 UTC m=+887.874061769" Oct 30 19:30:41.055863 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:41.054857 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t2t65" podStartSLOduration=2.085108752 podStartE2EDuration="5.054841994s" totalImagesPullingTime="2.969733242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:41.054646672 +0000 UTC m=+887.989434198" watchObservedRunningTime="2026-10-30 19:30:41.054841994 +0000 UTC m=+887.989629520" Oct 30 19:30:44.954278 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:44.954245 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-76877f96df-kbpcw"] Oct 30 19:30:44.957791 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:44.957774 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-76877f96df-kbpcw" Oct 30 19:30:45.009852 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:45.009832 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 19:30:45.059252 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:45.059229 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 19:30:45.059252 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:45.059229 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-lw9gz\"" Oct 30 19:30:45.060884 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:45.060866 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-76877f96df-kbpcw"] Oct 30 19:30:45.078948 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:45.078917 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljt9n\" (UniqueName: \"kubernetes.io/projected/adeb74f6-f548-4164-865b-0b8a2d1948b9-kube-api-access-ljt9n\") pod \"ai-gateway-controller-76877f96df-kbpcw\" (UID: \"adeb74f6-f548-4164-865b-0b8a2d1948b9\") " pod="opendatahub/ai-gateway-controller-76877f96df-kbpcw" Oct 30 19:30:45.079062 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:45.078954 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adeb74f6-f548-4164-865b-0b8a2d1948b9-metrics-certs\") pod \"ai-gateway-controller-76877f96df-kbpcw\" (UID: \"adeb74f6-f548-4164-865b-0b8a2d1948b9\") " pod="opendatahub/ai-gateway-controller-76877f96df-kbpcw" Oct 30 19:30:45.180333 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:45.180299 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljt9n\" (UniqueName: \"kubernetes.io/projected/adeb74f6-f548-4164-865b-0b8a2d1948b9-kube-api-access-ljt9n\") pod \"ai-gateway-controller-76877f96df-kbpcw\" (UID: \"adeb74f6-f548-4164-865b-0b8a2d1948b9\") " pod="opendatahub/ai-gateway-controller-76877f96df-kbpcw" Oct 30 19:30:45.180333 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:45.180333 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adeb74f6-f548-4164-865b-0b8a2d1948b9-metrics-certs\") pod \"ai-gateway-controller-76877f96df-kbpcw\" (UID: \"adeb74f6-f548-4164-865b-0b8a2d1948b9\") " pod="opendatahub/ai-gateway-controller-76877f96df-kbpcw" Oct 30 19:30:45.182847 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:45.182825 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adeb74f6-f548-4164-865b-0b8a2d1948b9-metrics-certs\") pod \"ai-gateway-controller-76877f96df-kbpcw\" (UID: \"adeb74f6-f548-4164-865b-0b8a2d1948b9\") " pod="opendatahub/ai-gateway-controller-76877f96df-kbpcw" Oct 30 19:30:45.237503 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:45.237439 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljt9n\" (UniqueName: \"kubernetes.io/projected/adeb74f6-f548-4164-865b-0b8a2d1948b9-kube-api-access-ljt9n\") pod \"ai-gateway-controller-76877f96df-kbpcw\" (UID: \"adeb74f6-f548-4164-865b-0b8a2d1948b9\") " pod="opendatahub/ai-gateway-controller-76877f96df-kbpcw" Oct 30 19:30:45.270094 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:45.270069 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-76877f96df-kbpcw" Oct 30 19:30:45.538070 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:45.538000 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:45.640880 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:45.640728 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-76877f96df-kbpcw"] Oct 30 19:30:45.643469 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:30:45.643440 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadeb74f6_f548_4164_865b_0b8a2d1948b9.slice/crio-cf6648240a019c08ff00a9cd66fe1fa33dcbc152674d51ca5561c61dbd815057 WatchSource:0}: Error finding container cf6648240a019c08ff00a9cd66fe1fa33dcbc152674d51ca5561c61dbd815057: Status 404 returned error can't find the container with id cf6648240a019c08ff00a9cd66fe1fa33dcbc152674d51ca5561c61dbd815057 Oct 30 19:30:45.852201 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:45.852170 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-76877f96df-kbpcw" event={"ID":"adeb74f6-f548-4164-865b-0b8a2d1948b9","Type":"ContainerStarted","Data":"cf6648240a019c08ff00a9cd66fe1fa33dcbc152674d51ca5561c61dbd815057"} Oct 30 19:30:46.695336 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:46.695303 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:48.868906 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:48.868871 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-76877f96df-kbpcw" event={"ID":"adeb74f6-f548-4164-865b-0b8a2d1948b9","Type":"ContainerStarted","Data":"ea8e942ad6af38683b3005c8a88a18f496acfef4e0a0c90cf8e72a68e5506e2a"} Oct 30 19:30:48.869311 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:48.868931 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-76877f96df-kbpcw" Oct 30 19:30:48.984820 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:48.984767 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-76877f96df-kbpcw" podStartSLOduration=2.167150334 podStartE2EDuration="4.984754259s" totalImagesPullingTime="2.817603925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:48.982891784 +0000 UTC m=+895.917679307" watchObservedRunningTime="2026-10-30 19:30:48.984754259 +0000 UTC m=+895.919541841" Oct 30 19:30:51.836002 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:51.835961 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t2t65" Oct 30 19:30:51.836385 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:51.836057 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-69g25" Oct 30 19:30:52.384074 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:52.384043 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:53.365269 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:30:53.365236 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:30:56.008947 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.008912 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk"] Oct 30 19:30:56.013597 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.013580 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.071837 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.071809 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.071969 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.071840 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.071969 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.071872 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.071969 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.071933 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.072112 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.072032 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.072112 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.072063 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.072112 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.072078 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.072112 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.072101 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef5e46b2-1008-4306-92fb-abfae43bcd79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.072248 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.072185 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt2fd\" (UniqueName: \"kubernetes.io/projected/ef5e46b2-1008-4306-92fb-abfae43bcd79-kube-api-access-bt2fd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.172798 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.172763 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.172798 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.172797 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.173017 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.172827 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.173017 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.172843 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.173017 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.172861 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.173017 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.172899 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.173017 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.172914 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.173017 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.172938 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef5e46b2-1008-4306-92fb-abfae43bcd79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.173017 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.172972 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt2fd\" (UniqueName: \"kubernetes.io/projected/ef5e46b2-1008-4306-92fb-abfae43bcd79-kube-api-access-bt2fd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.173417 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.173251 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.173417 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.173320 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.173506 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.173418 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.173506 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.173490 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.173872 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.173849 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef5e46b2-1008-4306-92fb-abfae43bcd79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.175075 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.175056 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.175380 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.175361 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.217389 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.217361 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk"] Oct 30 19:30:56.234222 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.234194 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.234332 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.234248 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt2fd\" (UniqueName: \"kubernetes.io/projected/ef5e46b2-1008-4306-92fb-abfae43bcd79-kube-api-access-bt2fd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r7gtk\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.325627 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.325599 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:56.748406 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.748384 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk"] Oct 30 19:30:56.750937 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:30:56.750904 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5e46b2_1008_4306_92fb_abfae43bcd79.slice/crio-b7e17b5d3034477ff779ed8580556defde0f06048941d92d2849866aaf936434 WatchSource:0}: Error finding container b7e17b5d3034477ff779ed8580556defde0f06048941d92d2849866aaf936434: Status 404 returned error can't find the container with id b7e17b5d3034477ff779ed8580556defde0f06048941d92d2849866aaf936434 Oct 30 19:30:56.753177 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.753147 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:56.753247 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.753207 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:56.753247 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.753236 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:56.907792 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.907756 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" event={"ID":"ef5e46b2-1008-4306-92fb-abfae43bcd79","Type":"ContainerStarted","Data":"2f12f6b78f24f7ceb865e861616b9fd2c9d5413956fd84c34b4244ef50e3bd17"} Oct 30 19:30:56.908004 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:56.907810 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" event={"ID":"ef5e46b2-1008-4306-92fb-abfae43bcd79","Type":"ContainerStarted","Data":"b7e17b5d3034477ff779ed8580556defde0f06048941d92d2849866aaf936434"} Oct 30 19:30:57.028350 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:57.028236 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" podStartSLOduration=2.028216628 podStartE2EDuration="2.028216628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:57.025318661 +0000 UTC m=+903.960106184" watchObservedRunningTime="2026-10-30 19:30:57.028216628 +0000 UTC m=+903.963004152" Oct 30 19:30:57.326725 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:57.326693 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:30:57.327749 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:57.327728 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:30:57.327837 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:57.327779 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 19:30:57.634364 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:57.634286 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:58.326293 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:58.326260 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:30:58.326761 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:58.326321 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 19:30:58.658786 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:58.658695 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:59.326261 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:59.326222 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:30:59.326429 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:59.326293 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 19:30:59.876505 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:30:59.876477 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-76877f96df-kbpcw" Oct 30 19:31:00.325920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:00.325891 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:31:00.326100 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:00.325940 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 19:31:01.326467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:01.326429 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:31:01.326959 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:01.326491 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 19:31:01.768367 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:01.768279 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:02.306017 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:02.302344 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:02.326932 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:02.326897 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:31:02.327337 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:02.326960 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 19:31:03.326897 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:03.326859 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:31:03.327105 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:03.326914 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 19:31:04.958630 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:04.958588 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:31:04.959096 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:04.959013 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:31:04.959742 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:04.959721 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 19:31:06.988662 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:06.988604 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:13.912587 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:13.912555 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:14.227198 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:14.227113 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:15.454772 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:15.454739 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:15.708411 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:15.708324 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:22.327785 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:22.327752 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:23.365124 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:31:23.365088 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:31:30.481867 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:30.481826 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:34.285075 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:34.285043 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:47.258566 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:31:47.258482 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:53.366509 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:31:53.366470 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:32:01.023602 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.023564 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:01.267955 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.267920 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:01.459425 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.459386 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj"] Oct 30 19:32:01.463191 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.463169 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.515583 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.515555 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-686689-openshift-default-dockercfg-75mcj\"" Oct 30 19:32:01.567149 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.567116 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj"] Oct 30 19:32:01.585508 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.585475 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk2gp\" (UniqueName: \"kubernetes.io/projected/3af9891c-2337-4531-a662-1eb43724b440-kube-api-access-jk2gp\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.585686 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.585542 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.585686 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.585585 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af9891c-2337-4531-a662-1eb43724b440-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.585686 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.585639 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af9891c-2337-4531-a662-1eb43724b440-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.585686 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.585659 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af9891c-2337-4531-a662-1eb43724b440-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.585910 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.585779 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.585910 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.585824 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.586035 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.585910 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.586035 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.585956 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.652953 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.652911 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:01.687271 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.687233 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk2gp\" (UniqueName: \"kubernetes.io/projected/3af9891c-2337-4531-a662-1eb43724b440-kube-api-access-jk2gp\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.687478 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.687295 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.687478 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.687410 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af9891c-2337-4531-a662-1eb43724b440-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.687478 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.687452 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af9891c-2337-4531-a662-1eb43724b440-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.687646 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.687480 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af9891c-2337-4531-a662-1eb43724b440-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.687646 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.687538 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.687646 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.687570 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.687646 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.687625 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.687828 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.687680 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.687914 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.687893 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.688007 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.687969 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.688102 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.688084 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.688155 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.688101 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.688350 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.688329 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af9891c-2337-4531-a662-1eb43724b440-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.690494 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.690473 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.690682 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.690665 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af9891c-2337-4531-a662-1eb43724b440-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.743885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.743811 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af9891c-2337-4531-a662-1eb43724b440-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.744077 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.744056 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk2gp\" (UniqueName: \"kubernetes.io/projected/3af9891c-2337-4531-a662-1eb43724b440-kube-api-access-jk2gp\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.775722 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.775687 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:01.913900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.913864 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:01.974041 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:01.973970 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" podUID="d17eaea73d7bf32a8a9bb9e63451f939" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:02.180577 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.180527 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:02.213533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.213501 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj"] Oct 30 19:32:02.216000 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:32:02.215956 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af9891c_2337_4531_a662_1eb43724b440.slice/crio-786e75557a435db71d837f1fff74a946467fb4b00057522280210d44f69621e2 WatchSource:0}: Error finding container 786e75557a435db71d837f1fff74a946467fb4b00057522280210d44f69621e2: Status 404 returned error can't find the container with id 786e75557a435db71d837f1fff74a946467fb4b00057522280210d44f69621e2 Oct 30 19:32:02.218886 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.218854 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:02.219036 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.218920 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:02.219036 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.218952 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:02.226382 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.226347 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" event={"ID":"3af9891c-2337-4531-a662-1eb43724b440","Type":"ContainerStarted","Data":"786e75557a435db71d837f1fff74a946467fb4b00057522280210d44f69621e2"} Oct 30 19:32:02.478255 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.478166 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx"] Oct 30 19:32:02.482013 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.481973 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.542655 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.542627 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5f55cd-openshift-default-dockercfg-wh68v\"" Oct 30 19:32:02.566366 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.566312 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx"] Oct 30 19:32:02.583746 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.583705 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:02.599171 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.599067 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-workload-certs\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.599394 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.599343 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-podinfo\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.599489 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.599415 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-token\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.599556 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.599492 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-credential-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.600051 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.600027 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh8zv\" (UniqueName: \"kubernetes.io/projected/f2042f53-156d-42c2-9554-5fe559fbe56f-kube-api-access-jh8zv\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.600169 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.600079 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-workload-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.600169 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.600105 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2042f53-156d-42c2-9554-5fe559fbe56f-istiod-ca-cert\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.600296 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.600195 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-data\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.600296 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.600268 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-envoy\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.701370 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.701318 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-workload-certs\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.701571 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.701405 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-podinfo\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.701571 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.701462 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-token\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.701571 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.701496 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-credential-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.701732 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.701577 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh8zv\" (UniqueName: \"kubernetes.io/projected/f2042f53-156d-42c2-9554-5fe559fbe56f-kube-api-access-jh8zv\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.701732 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.701622 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-workload-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.701732 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.701646 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2042f53-156d-42c2-9554-5fe559fbe56f-istiod-ca-cert\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.701893 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.701776 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-data\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.701893 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.701825 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-envoy\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.702012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.701935 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-credential-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.702247 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.702195 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-workload-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.702411 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.702386 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-data\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.702468 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.702420 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-workload-certs\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.702468 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.702425 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2042f53-156d-42c2-9554-5fe559fbe56f-istiod-ca-cert\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.704221 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.704191 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-envoy\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.704366 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.704255 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-podinfo\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.758503 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.758415 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-token\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.758503 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.758493 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh8zv\" (UniqueName: \"kubernetes.io/projected/f2042f53-156d-42c2-9554-5fe559fbe56f-kube-api-access-jh8zv\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.797634 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.797596 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:02.868034 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:02.861873 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:03.096529 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:03.096491 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:03.232477 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:03.232434 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx"] Oct 30 19:32:03.236160 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:03.236104 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" event={"ID":"3af9891c-2337-4531-a662-1eb43724b440","Type":"ContainerStarted","Data":"e65334778340ec7e024b969ec770140dc3783c3c5a2426d2cd1b1e42113c274f"} Oct 30 19:32:03.238011 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:32:03.237930 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2042f53_156d_42c2_9554_5fe559fbe56f.slice/crio-469ff7aa6eade33ca277ac6ad8b5045aaf135b803b2dae2714d13f08e748a10c WatchSource:0}: Error finding container 469ff7aa6eade33ca277ac6ad8b5045aaf135b803b2dae2714d13f08e748a10c: Status 404 returned error can't find the container with id 469ff7aa6eade33ca277ac6ad8b5045aaf135b803b2dae2714d13f08e748a10c Oct 30 19:32:03.241835 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:03.241789 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:03.241970 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:03.241891 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:03.241970 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:03.241935 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:03.357167 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:03.357044 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" podStartSLOduration=2.357021805 podStartE2EDuration="2.357021805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:03.354783532 +0000 UTC m=+970.289571057" watchObservedRunningTime="2026-10-30 19:32:03.357021805 +0000 UTC m=+970.291809351" Oct 30 19:32:03.777016 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:03.776884 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:03.782277 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:03.782247 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:04.244035 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:04.244001 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" event={"ID":"f2042f53-156d-42c2-9554-5fe559fbe56f","Type":"ContainerStarted","Data":"55e930b6f64912a78e94c389d42e9e1e413cd3c4b447eea781f0d496c6016fd3"} Oct 30 19:32:04.244547 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:04.244066 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" event={"ID":"f2042f53-156d-42c2-9554-5fe559fbe56f","Type":"ContainerStarted","Data":"469ff7aa6eade33ca277ac6ad8b5045aaf135b803b2dae2714d13f08e748a10c"} Oct 30 19:32:04.244547 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:04.244326 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:04.245549 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:04.245525 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:32:04.361610 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:04.361554 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" podStartSLOduration=2.361535063 podStartE2EDuration="2.361535063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:04.359665726 +0000 UTC m=+971.294453253" watchObservedRunningTime="2026-10-30 19:32:04.361535063 +0000 UTC m=+971.296322587" Oct 30 19:32:04.798246 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:04.798205 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:04.803788 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:04.803758 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:05.249766 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:05.249719 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:05.250616 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:05.250597 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:06.277215 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:06.277167 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:06.558216 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:06.558133 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:06.814827 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:06.814793 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:07.083830 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:07.083750 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:07.367786 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:07.367693 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:07.692495 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:07.692395 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:08.240234 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:08.240192 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:08.582277 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:08.582238 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:10.315191 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.315122 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:10.551465 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.551432 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:10.615223 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.615136 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh"] Oct 30 19:32:10.618806 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.618780 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" Oct 30 19:32:10.698859 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.698571 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4026634f-e0b5-4b64-a0ce-ebfe65c07365-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh\" (UID: \"4026634f-e0b5-4b64-a0ce-ebfe65c07365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" Oct 30 19:32:10.698859 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.698644 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqfzp\" (UniqueName: \"kubernetes.io/projected/4026634f-e0b5-4b64-a0ce-ebfe65c07365-kube-api-access-qqfzp\") pod \"payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh\" (UID: \"4026634f-e0b5-4b64-a0ce-ebfe65c07365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" Oct 30 19:32:10.719900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.719862 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-d52df0-dockercfg-rslnh\"" Oct 30 19:32:10.719900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.719854 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-d52df0\"" Oct 30 19:32:10.722033 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.722005 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh"] Oct 30 19:32:10.782319 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.782282 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft"] Oct 30 19:32:10.785920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.785887 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" Oct 30 19:32:10.799790 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.799759 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4026634f-e0b5-4b64-a0ce-ebfe65c07365-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh\" (UID: \"4026634f-e0b5-4b64-a0ce-ebfe65c07365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" Oct 30 19:32:10.799971 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.799803 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqfzp\" (UniqueName: \"kubernetes.io/projected/4026634f-e0b5-4b64-a0ce-ebfe65c07365-kube-api-access-qqfzp\") pod \"payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh\" (UID: \"4026634f-e0b5-4b64-a0ce-ebfe65c07365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" Oct 30 19:32:10.800613 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.800583 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4026634f-e0b5-4b64-a0ce-ebfe65c07365-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh\" (UID: \"4026634f-e0b5-4b64-a0ce-ebfe65c07365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" Oct 30 19:32:10.843739 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.843704 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft"] Oct 30 19:32:10.856678 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.856650 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqfzp\" (UniqueName: \"kubernetes.io/projected/4026634f-e0b5-4b64-a0ce-ebfe65c07365-kube-api-access-qqfzp\") pod \"payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh\" (UID: \"4026634f-e0b5-4b64-a0ce-ebfe65c07365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" Oct 30 19:32:10.901464 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.901314 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb823cde-6a63-4fcd-acc7-4700d65ba7ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft\" (UID: \"bb823cde-6a63-4fcd-acc7-4700d65ba7ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" Oct 30 19:32:10.901683 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.901477 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lxbc\" (UniqueName: \"kubernetes.io/projected/bb823cde-6a63-4fcd-acc7-4700d65ba7ac-kube-api-access-9lxbc\") pod \"payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft\" (UID: \"bb823cde-6a63-4fcd-acc7-4700d65ba7ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" Oct 30 19:32:10.932171 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:10.932135 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" Oct 30 19:32:11.002680 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:11.002643 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb823cde-6a63-4fcd-acc7-4700d65ba7ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft\" (UID: \"bb823cde-6a63-4fcd-acc7-4700d65ba7ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" Oct 30 19:32:11.002853 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:11.002749 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lxbc\" (UniqueName: \"kubernetes.io/projected/bb823cde-6a63-4fcd-acc7-4700d65ba7ac-kube-api-access-9lxbc\") pod \"payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft\" (UID: \"bb823cde-6a63-4fcd-acc7-4700d65ba7ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" Oct 30 19:32:11.003412 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:11.003385 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb823cde-6a63-4fcd-acc7-4700d65ba7ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft\" (UID: \"bb823cde-6a63-4fcd-acc7-4700d65ba7ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" Oct 30 19:32:11.032589 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:11.032551 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:11.060076 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:11.060044 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lxbc\" (UniqueName: \"kubernetes.io/projected/bb823cde-6a63-4fcd-acc7-4700d65ba7ac-kube-api-access-9lxbc\") pod \"payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft\" (UID: \"bb823cde-6a63-4fcd-acc7-4700d65ba7ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" Oct 30 19:32:11.100549 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:11.100511 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" Oct 30 19:32:11.309307 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:11.309282 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh"] Oct 30 19:32:11.311925 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:32:11.311889 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4026634f_e0b5_4b64_a0ce_ebfe65c07365.slice/crio-6db5e2cdb5964aeb26ee4c4bfdf7f56e7fc15fea0455fd69e2adb2f04226cdbd WatchSource:0}: Error finding container 6db5e2cdb5964aeb26ee4c4bfdf7f56e7fc15fea0455fd69e2adb2f04226cdbd: Status 404 returned error can't find the container with id 6db5e2cdb5964aeb26ee4c4bfdf7f56e7fc15fea0455fd69e2adb2f04226cdbd Oct 30 19:32:11.471387 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:11.471358 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft"] Oct 30 19:32:11.474201 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:32:11.474169 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb823cde_6a63_4fcd_acc7_4700d65ba7ac.slice/crio-9d8fbcf7407fd635aa33d1a7a6f953d923cd2e2412b3e8545d6214401224c89d WatchSource:0}: Error finding container 9d8fbcf7407fd635aa33d1a7a6f953d923cd2e2412b3e8545d6214401224c89d: Status 404 returned error can't find the container with id 9d8fbcf7407fd635aa33d1a7a6f953d923cd2e2412b3e8545d6214401224c89d Oct 30 19:32:11.638163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:11.638128 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:11.868498 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:11.868462 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:12.229014 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:12.228899 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:12.289816 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:12.289772 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" event={"ID":"bb823cde-6a63-4fcd-acc7-4700d65ba7ac","Type":"ContainerStarted","Data":"0c127ed7314151428769ac31cc3b831da26cb96ce74beb382a4debb5a8c7c0cb"} Oct 30 19:32:12.290048 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:12.289873 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" event={"ID":"bb823cde-6a63-4fcd-acc7-4700d65ba7ac","Type":"ContainerStarted","Data":"9d8fbcf7407fd635aa33d1a7a6f953d923cd2e2412b3e8545d6214401224c89d"} Oct 30 19:32:12.290048 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:12.289907 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" Oct 30 19:32:12.291387 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:12.291353 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" event={"ID":"4026634f-e0b5-4b64-a0ce-ebfe65c07365","Type":"ContainerStarted","Data":"6d43f4b52e7b7ca99cc7ca6ee6c337c411c8721dcacbae58e8a400fefd64ab6d"} Oct 30 19:32:12.291517 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:12.291403 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" event={"ID":"4026634f-e0b5-4b64-a0ce-ebfe65c07365","Type":"ContainerStarted","Data":"6db5e2cdb5964aeb26ee4c4bfdf7f56e7fc15fea0455fd69e2adb2f04226cdbd"} Oct 30 19:32:12.291517 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:12.291457 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" Oct 30 19:32:12.411541 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:12.411477 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" podStartSLOduration=2.411460808 podStartE2EDuration="2.411460808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:12.409176458 +0000 UTC m=+979.343964046" watchObservedRunningTime="2026-10-30 19:32:12.411460808 +0000 UTC m=+979.346248345" Oct 30 19:32:12.484618 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:12.484519 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:12.526146 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:12.526074 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" podStartSLOduration=2.526053351 podStartE2EDuration="2.526053351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:12.522913454 +0000 UTC m=+979.457700978" watchObservedRunningTime="2026-10-30 19:32:12.526053351 +0000 UTC m=+979.460840879" Oct 30 19:32:12.711636 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:12.711605 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:12.948546 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:12.948516 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:15.552059 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:15.552026 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:15.769255 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:15.769208 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:15.878411 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:15.878375 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt"] Oct 30 19:32:15.882100 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:15.882068 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" Oct 30 19:32:15.888866 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:15.888653 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edac81ed-b1b2-4ddb-a2ae-bfe897991191-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt\" (UID: \"edac81ed-b1b2-4ddb-a2ae-bfe897991191\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" Oct 30 19:32:15.888866 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:15.888759 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs569\" (UniqueName: \"kubernetes.io/projected/edac81ed-b1b2-4ddb-a2ae-bfe897991191-kube-api-access-fs569\") pod \"payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt\" (UID: \"edac81ed-b1b2-4ddb-a2ae-bfe897991191\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" Oct 30 19:32:15.933766 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:15.933727 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-ada8d7\"" Oct 30 19:32:15.983315 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:15.983277 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-ada8d7-dockercfg-zx8j5\"" Oct 30 19:32:15.986551 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:15.986521 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt"] Oct 30 19:32:15.989337 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:15.989315 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs569\" (UniqueName: \"kubernetes.io/projected/edac81ed-b1b2-4ddb-a2ae-bfe897991191-kube-api-access-fs569\") pod \"payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt\" (UID: \"edac81ed-b1b2-4ddb-a2ae-bfe897991191\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" Oct 30 19:32:15.989417 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:15.989406 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edac81ed-b1b2-4ddb-a2ae-bfe897991191-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt\" (UID: \"edac81ed-b1b2-4ddb-a2ae-bfe897991191\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" Oct 30 19:32:15.990072 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:15.990053 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edac81ed-b1b2-4ddb-a2ae-bfe897991191-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt\" (UID: \"edac81ed-b1b2-4ddb-a2ae-bfe897991191\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" Oct 30 19:32:16.034704 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:16.034665 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb"] Oct 30 19:32:16.038764 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:16.038735 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" Oct 30 19:32:16.049013 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:16.048949 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs569\" (UniqueName: \"kubernetes.io/projected/edac81ed-b1b2-4ddb-a2ae-bfe897991191-kube-api-access-fs569\") pod \"payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt\" (UID: \"edac81ed-b1b2-4ddb-a2ae-bfe897991191\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" Oct 30 19:32:16.084472 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:16.084430 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb"] Oct 30 19:32:16.090215 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:16.090176 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb\" (UID: \"c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" Oct 30 19:32:16.090397 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:16.090365 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xg6d\" (UniqueName: \"kubernetes.io/projected/c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904-kube-api-access-9xg6d\") pod \"payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb\" (UID: \"c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" Oct 30 19:32:16.191744 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:16.191647 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xg6d\" (UniqueName: \"kubernetes.io/projected/c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904-kube-api-access-9xg6d\") pod \"payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb\" (UID: \"c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" Oct 30 19:32:16.191744 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:16.191733 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb\" (UID: \"c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" Oct 30 19:32:16.192686 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:16.192648 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb\" (UID: \"c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" Oct 30 19:32:16.196567 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:16.196533 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" Oct 30 19:32:16.250078 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:16.249704 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xg6d\" (UniqueName: \"kubernetes.io/projected/c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904-kube-api-access-9xg6d\") pod \"payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb\" (UID: \"c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" Oct 30 19:32:16.367499 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:16.367455 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" Oct 30 19:32:16.574494 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:16.574466 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt"] Oct 30 19:32:16.575905 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:32:16.575876 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedac81ed_b1b2_4ddb_a2ae_bfe897991191.slice/crio-6330b82928a3c9ebe0e2ce2699fb3abd07ec8ceb08f35786ae2cd238fdc0b125 WatchSource:0}: Error finding container 6330b82928a3c9ebe0e2ce2699fb3abd07ec8ceb08f35786ae2cd238fdc0b125: Status 404 returned error can't find the container with id 6330b82928a3c9ebe0e2ce2699fb3abd07ec8ceb08f35786ae2cd238fdc0b125 Oct 30 19:32:16.740867 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:16.740840 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb"] Oct 30 19:32:16.742849 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:32:16.742819 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f4e9af_3d6f_499c_a4b8_a5e8c47fd904.slice/crio-e73502aa52ea42c3216c164e334e7f4a8aed9271872b457499a78aaa1b8cc110 WatchSource:0}: Error finding container e73502aa52ea42c3216c164e334e7f4a8aed9271872b457499a78aaa1b8cc110: Status 404 returned error can't find the container with id e73502aa52ea42c3216c164e334e7f4a8aed9271872b457499a78aaa1b8cc110 Oct 30 19:32:17.318462 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.318423 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" event={"ID":"c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904","Type":"ContainerStarted","Data":"dec892ca2f8293c803cc61367f0c51afc33e8ecfb35f939fa37e9cf3cf684b3f"} Oct 30 19:32:17.318699 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.318487 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" event={"ID":"c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904","Type":"ContainerStarted","Data":"e73502aa52ea42c3216c164e334e7f4a8aed9271872b457499a78aaa1b8cc110"} Oct 30 19:32:17.318699 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.318534 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" Oct 30 19:32:17.320177 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.320154 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" event={"ID":"edac81ed-b1b2-4ddb-a2ae-bfe897991191","Type":"ContainerStarted","Data":"1d8fe6454a3d87a0b050c54a7ce2119117ecebc4d202e0746e7322c8316c8fc2"} Oct 30 19:32:17.320297 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.320191 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" event={"ID":"edac81ed-b1b2-4ddb-a2ae-bfe897991191","Type":"ContainerStarted","Data":"6330b82928a3c9ebe0e2ce2699fb3abd07ec8ceb08f35786ae2cd238fdc0b125"} Oct 30 19:32:17.320297 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.320251 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" Oct 30 19:32:17.338588 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.338560 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj"] Oct 30 19:32:17.342199 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.342184 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:17.395339 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.395306 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-5f55cd-serving-cert\"" Oct 30 19:32:17.403925 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.403889 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:17.404088 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.403933 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxjk7\" (UniqueName: \"kubernetes.io/projected/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-kube-api-access-fxjk7\") pod \"maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:17.404135 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.404117 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:17.404288 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.404262 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-metrics-certs\") pod \"maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:17.410390 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.410362 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj"] Oct 30 19:32:17.460319 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.460266 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" podStartSLOduration=2.460250901 podStartE2EDuration="2.460250901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:17.458575565 +0000 UTC m=+984.393363091" watchObservedRunningTime="2026-10-30 19:32:17.460250901 +0000 UTC m=+984.395038420" Oct 30 19:32:17.505419 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.505370 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxjk7\" (UniqueName: \"kubernetes.io/projected/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-kube-api-access-fxjk7\") pod \"maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:17.505599 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.505469 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:17.505599 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.505554 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-metrics-certs\") pod \"maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:17.505720 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.505632 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:17.506192 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:32:17.505886 4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-serving-cert: secret "maas-api-e2e-worker-w6-5f55cd-serving-cert" not found Oct 30 19:32:17.506192 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:32:17.505962 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-maas-api-tls podName:aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0 nodeName:}" failed. No retries permitted until 2026-10-30 19:32:18.00594316 +0000 UTC m=+984.940730671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-maas-api-tls") pod "maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" (UID: "aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0") : secret "maas-api-e2e-worker-w6-5f55cd-serving-cert" not found Oct 30 19:32:17.506545 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.506523 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:17.508445 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.508420 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-metrics-certs\") pod \"maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:17.574997 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.574910 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxjk7\" (UniqueName: \"kubernetes.io/projected/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-kube-api-access-fxjk7\") pod \"maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:17.591679 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.591619 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" podStartSLOduration=2.591600165 podStartE2EDuration="2.591600165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:17.590097882 +0000 UTC m=+984.524885407" watchObservedRunningTime="2026-10-30 19:32:17.591600165 +0000 UTC m=+984.526387692" Oct 30 19:32:17.758541 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:17.758510 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:18.004771 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.004689 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:18.009934 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.009912 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:18.012404 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.012373 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:18.256250 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.256152 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:18.330763 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.330731 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:18.593450 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.593417 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh"] Oct 30 19:32:18.598151 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.598123 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" Oct 30 19:32:18.619176 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.619146 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vvcz\" (UniqueName: \"kubernetes.io/projected/e60baa7a-9536-4dd5-b59c-d042cd41b798-kube-api-access-4vvcz\") pod \"payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh\" (UID: \"e60baa7a-9536-4dd5-b59c-d042cd41b798\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" Oct 30 19:32:18.619301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.619187 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60baa7a-9536-4dd5-b59c-d042cd41b798-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh\" (UID: \"e60baa7a-9536-4dd5-b59c-d042cd41b798\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" Oct 30 19:32:18.653503 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.653458 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj"] Oct 30 19:32:18.654060 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.654046 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-686689\"" Oct 30 19:32:18.656656 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:32:18.656618 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa49aae0_aa9c_43e4_a3cd_d27cb46a5fd0.slice/crio-605ef0eda04283c854413a70c952d433cf0ec17ed0e4d2af5d0df35d57695f2f WatchSource:0}: Error finding container 605ef0eda04283c854413a70c952d433cf0ec17ed0e4d2af5d0df35d57695f2f: Status 404 returned error can't find the container with id 605ef0eda04283c854413a70c952d433cf0ec17ed0e4d2af5d0df35d57695f2f Oct 30 19:32:18.658768 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.658749 4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:32:18.659096 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.659077 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh"] Oct 30 19:32:18.670411 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.670385 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:18.734806 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.731969 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvcz\" (UniqueName: \"kubernetes.io/projected/e60baa7a-9536-4dd5-b59c-d042cd41b798-kube-api-access-4vvcz\") pod \"payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh\" (UID: \"e60baa7a-9536-4dd5-b59c-d042cd41b798\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" Oct 30 19:32:18.734806 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.732055 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60baa7a-9536-4dd5-b59c-d042cd41b798-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh\" (UID: \"e60baa7a-9536-4dd5-b59c-d042cd41b798\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" Oct 30 19:32:18.734806 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.733895 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60baa7a-9536-4dd5-b59c-d042cd41b798-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh\" (UID: \"e60baa7a-9536-4dd5-b59c-d042cd41b798\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" Oct 30 19:32:18.782992 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.782959 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd"] Oct 30 19:32:18.787157 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.787137 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" Oct 30 19:32:18.793067 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.793041 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vvcz\" (UniqueName: \"kubernetes.io/projected/e60baa7a-9536-4dd5-b59c-d042cd41b798-kube-api-access-4vvcz\") pod \"payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh\" (UID: \"e60baa7a-9536-4dd5-b59c-d042cd41b798\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" Oct 30 19:32:18.833205 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.833169 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6lkf\" (UniqueName: \"kubernetes.io/projected/34f8f88e-3537-4e62-ae87-b12ec3812f62-kube-api-access-p6lkf\") pod \"payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd\" (UID: \"34f8f88e-3537-4e62-ae87-b12ec3812f62\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" Oct 30 19:32:18.833356 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.833325 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34f8f88e-3537-4e62-ae87-b12ec3812f62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd\" (UID: \"34f8f88e-3537-4e62-ae87-b12ec3812f62\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" Oct 30 19:32:18.848845 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.848755 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-686689-dockercfg-74l6k\"" Oct 30 19:32:18.848845 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.848756 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd"] Oct 30 19:32:18.911910 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.911873 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" Oct 30 19:32:18.934704 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.934672 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6lkf\" (UniqueName: \"kubernetes.io/projected/34f8f88e-3537-4e62-ae87-b12ec3812f62-kube-api-access-p6lkf\") pod \"payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd\" (UID: \"34f8f88e-3537-4e62-ae87-b12ec3812f62\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" Oct 30 19:32:18.934872 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.934808 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34f8f88e-3537-4e62-ae87-b12ec3812f62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd\" (UID: \"34f8f88e-3537-4e62-ae87-b12ec3812f62\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" Oct 30 19:32:18.935599 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.935575 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34f8f88e-3537-4e62-ae87-b12ec3812f62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd\" (UID: \"34f8f88e-3537-4e62-ae87-b12ec3812f62\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" Oct 30 19:32:18.992924 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:18.992890 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6lkf\" (UniqueName: \"kubernetes.io/projected/34f8f88e-3537-4e62-ae87-b12ec3812f62-kube-api-access-p6lkf\") pod \"payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd\" (UID: \"34f8f88e-3537-4e62-ae87-b12ec3812f62\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" Oct 30 19:32:19.111042 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:19.110915 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" Oct 30 19:32:19.288885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:19.288857 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh"] Oct 30 19:32:19.291452 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:32:19.291423 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60baa7a_9536_4dd5_b59c_d042cd41b798.slice/crio-10e3e7e4dcf2a133e9d2c59acf7f8038ab093bfaf114b279cc4a83fa1ca2f698 WatchSource:0}: Error finding container 10e3e7e4dcf2a133e9d2c59acf7f8038ab093bfaf114b279cc4a83fa1ca2f698: Status 404 returned error can't find the container with id 10e3e7e4dcf2a133e9d2c59acf7f8038ab093bfaf114b279cc4a83fa1ca2f698 Oct 30 19:32:19.312369 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:19.312341 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:19.333246 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:19.333213 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" event={"ID":"e60baa7a-9536-4dd5-b59c-d042cd41b798","Type":"ContainerStarted","Data":"10e3e7e4dcf2a133e9d2c59acf7f8038ab093bfaf114b279cc4a83fa1ca2f698"} Oct 30 19:32:19.334412 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:19.334380 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" event={"ID":"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0","Type":"ContainerStarted","Data":"605ef0eda04283c854413a70c952d433cf0ec17ed0e4d2af5d0df35d57695f2f"} Oct 30 19:32:19.493734 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:19.493703 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd"] Oct 30 19:32:19.496538 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:32:19.496506 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f8f88e_3537_4e62_ae87_b12ec3812f62.slice/crio-682e8522b2c2c24988a6bd20a1ff0641d1130db10dea33e4ce205658f432afd7 WatchSource:0}: Error finding container 682e8522b2c2c24988a6bd20a1ff0641d1130db10dea33e4ce205658f432afd7: Status 404 returned error can't find the container with id 682e8522b2c2c24988a6bd20a1ff0641d1130db10dea33e4ce205658f432afd7 Oct 30 19:32:19.616824 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:19.616755 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:19.907345 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:19.907264 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:20.171009 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:20.170905 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:20.341850 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:20.341812 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" event={"ID":"e60baa7a-9536-4dd5-b59c-d042cd41b798","Type":"ContainerStarted","Data":"7aa07c6e89eafcb90edfce328f8c892e08d8552e831a34b78f005fb76b0af478"} Oct 30 19:32:20.342105 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:20.341968 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" Oct 30 19:32:20.343370 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:20.343343 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" event={"ID":"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0","Type":"ContainerStarted","Data":"9a9855312bc9447b2d0bba4c54b416af520d5a6ba2552e5036c66f0dca873df8"} Oct 30 19:32:20.343467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:20.343400 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:20.344799 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:20.344779 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" event={"ID":"34f8f88e-3537-4e62-ae87-b12ec3812f62","Type":"ContainerStarted","Data":"b03cdc10b9b3c337f6ee33f18f184de3843bff71f25c515def64405888e84828"} Oct 30 19:32:20.344870 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:20.344822 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" event={"ID":"34f8f88e-3537-4e62-ae87-b12ec3812f62","Type":"ContainerStarted","Data":"682e8522b2c2c24988a6bd20a1ff0641d1130db10dea33e4ce205658f432afd7"} Oct 30 19:32:20.344870 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:20.344841 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" Oct 30 19:32:20.475012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:20.474890 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" podStartSLOduration=2.474872491 podStartE2EDuration="2.474872491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:20.472568266 +0000 UTC m=+987.407355791" watchObservedRunningTime="2026-10-30 19:32:20.474872491 +0000 UTC m=+987.409660018" Oct 30 19:32:20.596255 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:20.596200 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" podStartSLOduration=2.715870129 podStartE2EDuration="3.596184705s" totalImagesPullingTime="880.314576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:20.593202515 +0000 UTC m=+987.527990043" watchObservedRunningTime="2026-10-30 19:32:20.596184705 +0000 UTC m=+987.530972235" Oct 30 19:32:20.713859 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:20.708688 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" podStartSLOduration=2.708666059 podStartE2EDuration="2.708666059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:20.707582385 +0000 UTC m=+987.642369911" watchObservedRunningTime="2026-10-30 19:32:20.708666059 +0000 UTC m=+987.643453579" Oct 30 19:32:21.097392 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:21.097359 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:21.340338 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:21.340294 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:21.971809 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:21.971772 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:22.203268 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.203234 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:22.657081 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.657047 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:22.667800 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.667767 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw"] Oct 30 19:32:22.671676 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.671651 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:22.723555 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.723521 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-d52df0-serving-cert\"" Oct 30 19:32:22.774360 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.774324 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-metrics-certs\") pod \"maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:22.774535 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.774366 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:22.774535 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.774492 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:22.774676 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.774656 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84bn4\" (UniqueName: \"kubernetes.io/projected/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-kube-api-access-84bn4\") pod \"maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:22.775427 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.775402 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw"] Oct 30 19:32:22.876196 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.876158 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84bn4\" (UniqueName: \"kubernetes.io/projected/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-kube-api-access-84bn4\") pod \"maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:22.876354 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.876320 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-metrics-certs\") pod \"maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:22.876456 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.876366 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:22.876456 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.876428 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:22.876539 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:32:22.876514 4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-serving-cert: secret "maas-api-e2e-worker-w1-d52df0-serving-cert" not found Oct 30 19:32:22.876636 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:32:22.876601 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-maas-api-tls podName:48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9 nodeName:}" failed. No retries permitted until 2026-10-30 19:32:23.376579313 +0000 UTC m=+990.311366822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-maas-api-tls") pod "maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" (UID: "48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9") : secret "maas-api-e2e-worker-w1-d52df0-serving-cert" not found Oct 30 19:32:22.877175 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.877157 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:22.878749 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.878727 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-metrics-certs\") pod \"maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:22.933582 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.933497 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84bn4\" (UniqueName: \"kubernetes.io/projected/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-kube-api-access-84bn4\") pod \"maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:22.989735 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:22.989703 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:23.210296 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:23.210166 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:23.299820 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:23.299789 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" Oct 30 19:32:23.300023 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:23.299857 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" Oct 30 19:32:23.374141 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:32:23.374101 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:32:23.383755 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:23.383715 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:23.386312 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:23.386284 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:23.585988 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:23.585930 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:24.016188 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:24.016160 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw"] Oct 30 19:32:24.018416 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:32:24.018386 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a83cf5_b6bb_4b67_ab3c_d4c0c42c9fc9.slice/crio-bc550908ecb1741d0df4d1c2bacdfbd12a9918e186f09dd22216631e8ca15429 WatchSource:0}: Error finding container bc550908ecb1741d0df4d1c2bacdfbd12a9918e186f09dd22216631e8ca15429: Status 404 returned error can't find the container with id bc550908ecb1741d0df4d1c2bacdfbd12a9918e186f09dd22216631e8ca15429 Oct 30 19:32:24.367174 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:24.367137 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" event={"ID":"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9","Type":"ContainerStarted","Data":"bc550908ecb1741d0df4d1c2bacdfbd12a9918e186f09dd22216631e8ca15429"} Oct 30 19:32:25.374311 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:25.374264 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" event={"ID":"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9","Type":"ContainerStarted","Data":"09568bd15df887959b0afc56a5e07b7b10a4ef36d8d32abae9afe1a11e54a7e6"} Oct 30 19:32:25.374767 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:25.374345 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:25.491112 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:25.491050 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" podStartSLOduration=2.717050767 podStartE2EDuration="3.49103469s" totalImagesPullingTime="773.983923ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:25.490518908 +0000 UTC m=+992.425306452" watchObservedRunningTime="2026-10-30 19:32:25.49103469 +0000 UTC m=+992.425822214" Oct 30 19:32:26.355863 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:26.355824 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 19:32:26.683668 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:26.683568 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:26.951582 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:26.951496 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:28.331698 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:28.331668 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" Oct 30 19:32:28.332171 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:28.331890 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" Oct 30 19:32:28.468233 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:28.468195 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:31.353235 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:31.353197 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" Oct 30 19:32:31.353692 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:31.353375 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" Oct 30 19:32:31.410071 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:31.410034 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:31.410306 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:31.410286 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:32:31.631206 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:31.631057 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:31.880054 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:31.880009 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7"] Oct 30 19:32:31.886289 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:31.886203 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" Oct 30 19:32:31.940468 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:31.940410 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-a3f9b7\"" Oct 30 19:32:31.941519 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:31.941490 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7"] Oct 30 19:32:31.941819 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:31.941792 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-a3f9b7-dockercfg-cs4qp\"" Oct 30 19:32:31.990737 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:31.990694 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr8td\" (UniqueName: \"kubernetes.io/projected/23196d8b-4348-4e50-9f54-5c44c4b6b35c-kube-api-access-xr8td\") pod \"payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7\" (UID: \"23196d8b-4348-4e50-9f54-5c44c4b6b35c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" Oct 30 19:32:31.990898 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:31.990746 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23196d8b-4348-4e50-9f54-5c44c4b6b35c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7\" (UID: \"23196d8b-4348-4e50-9f54-5c44c4b6b35c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" Oct 30 19:32:32.092275 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:32.092210 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr8td\" (UniqueName: \"kubernetes.io/projected/23196d8b-4348-4e50-9f54-5c44c4b6b35c-kube-api-access-xr8td\") pod \"payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7\" (UID: \"23196d8b-4348-4e50-9f54-5c44c4b6b35c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" Oct 30 19:32:32.092478 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:32.092330 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23196d8b-4348-4e50-9f54-5c44c4b6b35c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7\" (UID: \"23196d8b-4348-4e50-9f54-5c44c4b6b35c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" Oct 30 19:32:32.093315 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:32.093290 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23196d8b-4348-4e50-9f54-5c44c4b6b35c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7\" (UID: \"23196d8b-4348-4e50-9f54-5c44c4b6b35c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" Oct 30 19:32:32.149479 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:32.149397 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr8td\" (UniqueName: \"kubernetes.io/projected/23196d8b-4348-4e50-9f54-5c44c4b6b35c-kube-api-access-xr8td\") pod \"payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7\" (UID: \"23196d8b-4348-4e50-9f54-5c44c4b6b35c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" Oct 30 19:32:32.203565 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:32.203517 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" Oct 30 19:32:32.588244 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:32.588208 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7"] Oct 30 19:32:32.590591 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:32:32.590556 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23196d8b_4348_4e50_9f54_5c44c4b6b35c.slice/crio-e805afccf88dc7703fbcbb8d4290887f2615ba57fa5d3e957f1a8d0dfc923543 WatchSource:0}: Error finding container e805afccf88dc7703fbcbb8d4290887f2615ba57fa5d3e957f1a8d0dfc923543: Status 404 returned error can't find the container with id e805afccf88dc7703fbcbb8d4290887f2615ba57fa5d3e957f1a8d0dfc923543 Oct 30 19:32:33.445918 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:33.445870 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" event={"ID":"23196d8b-4348-4e50-9f54-5c44c4b6b35c","Type":"ContainerStarted","Data":"76f3117e773731c930e8598876b7bf56cc0c59a035332a34ef139c5149ff525d"} Oct 30 19:32:33.446138 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:33.445949 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" Oct 30 19:32:33.446138 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:33.445994 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" event={"ID":"23196d8b-4348-4e50-9f54-5c44c4b6b35c","Type":"ContainerStarted","Data":"e805afccf88dc7703fbcbb8d4290887f2615ba57fa5d3e957f1a8d0dfc923543"} Oct 30 19:32:33.561124 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:33.560784 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" podStartSLOduration=2.560767057 podStartE2EDuration="2.560767057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:33.559950848 +0000 UTC m=+1000.494738370" watchObservedRunningTime="2026-10-30 19:32:33.560767057 +0000 UTC m=+1000.495554583" Oct 30 19:32:33.581614 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:33.581551 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:35.251512 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:35.251478 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 19:32:35.251949 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:35.251546 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" podUID="f2042f53-156d-42c2-9554-5fe559fbe56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 19:32:35.463920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:35.463808 4433 generic.go:356] "Generic (PLEG): container finished" podID="f2042f53-156d-42c2-9554-5fe559fbe56f" containerID="55e930b6f64912a78e94c389d42e9e1e413cd3c4b447eea781f0d496c6016fd3" exitCode=0 Oct 30 19:32:35.463920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:35.463892 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" event={"ID":"f2042f53-156d-42c2-9554-5fe559fbe56f","Type":"ContainerDied","Data":"55e930b6f64912a78e94c389d42e9e1e413cd3c4b447eea781f0d496c6016fd3"} Oct 30 19:32:35.464342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:35.464326 4433 scope.go:122] "RemoveContainer" containerID="55e930b6f64912a78e94c389d42e9e1e413cd3c4b447eea781f0d496c6016fd3" Oct 30 19:32:35.465752 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:35.465726 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:35.465812 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:35.465790 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:35.465846 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:35.465823 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:36.473575 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:36.473539 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" event={"ID":"f2042f53-156d-42c2-9554-5fe559fbe56f","Type":"ContainerStarted","Data":"a7f3bcbd9ebc63079f1e74772b837350830b7c74bad2545cc97dbdaf8e120083"} Oct 30 19:32:36.534525 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:36.534494 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:36.768362 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:36.768281 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:36.802694 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:36.802653 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:36.803114 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:36.803029 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 19:32:36.803310 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:36.803285 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" podUID="f2042f53-156d-42c2-9554-5fe559fbe56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 19:32:37.803005 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:37.802396 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 19:32:37.803005 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:37.802462 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" podUID="f2042f53-156d-42c2-9554-5fe559fbe56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 19:32:38.551451 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:38.551415 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:38.801022 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:38.800989 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 19:32:38.801291 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:38.801253 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" podUID="f2042f53-156d-42c2-9554-5fe559fbe56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 19:32:38.801941 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:38.801862 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:39.030783 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.030751 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:39.383073 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.383042 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:39.622931 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.622898 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:39.717441 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.717354 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t"] Oct 30 19:32:39.720076 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.720058 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" Oct 30 19:32:39.772632 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.772590 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7xqf\" (UniqueName: \"kubernetes.io/projected/5ecee9ae-a839-4edc-b06b-cc014cb7e7ff-kube-api-access-g7xqf\") pod \"payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t\" (UID: \"5ecee9ae-a839-4edc-b06b-cc014cb7e7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" Oct 30 19:32:39.772785 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.772759 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecee9ae-a839-4edc-b06b-cc014cb7e7ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t\" (UID: \"5ecee9ae-a839-4edc-b06b-cc014cb7e7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" Oct 30 19:32:39.817365 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.817328 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 19:32:39.826174 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.826132 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" podUID="f2042f53-156d-42c2-9554-5fe559fbe56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 19:32:39.830123 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.830093 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b5178f\"" Oct 30 19:32:39.830378 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.830359 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b5178f-dockercfg-2f6r7\"" Oct 30 19:32:39.834655 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.833409 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t"] Oct 30 19:32:39.874298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.874253 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecee9ae-a839-4edc-b06b-cc014cb7e7ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t\" (UID: \"5ecee9ae-a839-4edc-b06b-cc014cb7e7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" Oct 30 19:32:39.874464 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.874369 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7xqf\" (UniqueName: \"kubernetes.io/projected/5ecee9ae-a839-4edc-b06b-cc014cb7e7ff-kube-api-access-g7xqf\") pod \"payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t\" (UID: \"5ecee9ae-a839-4edc-b06b-cc014cb7e7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" Oct 30 19:32:39.874926 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.874906 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecee9ae-a839-4edc-b06b-cc014cb7e7ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t\" (UID: \"5ecee9ae-a839-4edc-b06b-cc014cb7e7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" Oct 30 19:32:39.878959 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.878934 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd"] Oct 30 19:32:39.881392 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.881375 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" Oct 30 19:32:39.932855 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.932824 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd"] Oct 30 19:32:39.934102 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.934076 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7xqf\" (UniqueName: \"kubernetes.io/projected/5ecee9ae-a839-4edc-b06b-cc014cb7e7ff-kube-api-access-g7xqf\") pod \"payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t\" (UID: \"5ecee9ae-a839-4edc-b06b-cc014cb7e7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" Oct 30 19:32:39.975792 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.975724 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f54b26b9-651b-4ba4-82a4-57819b4f02b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd\" (UID: \"f54b26b9-651b-4ba4-82a4-57819b4f02b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" Oct 30 19:32:39.975792 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:39.975757 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjnhb\" (UniqueName: \"kubernetes.io/projected/f54b26b9-651b-4ba4-82a4-57819b4f02b4-kube-api-access-tjnhb\") pod \"payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd\" (UID: \"f54b26b9-651b-4ba4-82a4-57819b4f02b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" Oct 30 19:32:40.032022 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:40.031972 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" Oct 30 19:32:40.077148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:40.077115 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f54b26b9-651b-4ba4-82a4-57819b4f02b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd\" (UID: \"f54b26b9-651b-4ba4-82a4-57819b4f02b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" Oct 30 19:32:40.077292 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:40.077158 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjnhb\" (UniqueName: \"kubernetes.io/projected/f54b26b9-651b-4ba4-82a4-57819b4f02b4-kube-api-access-tjnhb\") pod \"payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd\" (UID: \"f54b26b9-651b-4ba4-82a4-57819b4f02b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" Oct 30 19:32:40.077851 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:40.077831 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f54b26b9-651b-4ba4-82a4-57819b4f02b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd\" (UID: \"f54b26b9-651b-4ba4-82a4-57819b4f02b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" Oct 30 19:32:40.135049 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:40.134997 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjnhb\" (UniqueName: \"kubernetes.io/projected/f54b26b9-651b-4ba4-82a4-57819b4f02b4-kube-api-access-tjnhb\") pod \"payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd\" (UID: \"f54b26b9-651b-4ba4-82a4-57819b4f02b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" Oct 30 19:32:40.215332 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:40.215295 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" Oct 30 19:32:40.412624 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:40.412591 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t"] Oct 30 19:32:40.414263 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:32:40.414237 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecee9ae_a839_4edc_b06b_cc014cb7e7ff.slice/crio-58f05fc19823e83257cf23e8664a5fff57fefdde0af8ab634f95cfb32027d89d WatchSource:0}: Error finding container 58f05fc19823e83257cf23e8664a5fff57fefdde0af8ab634f95cfb32027d89d: Status 404 returned error can't find the container with id 58f05fc19823e83257cf23e8664a5fff57fefdde0af8ab634f95cfb32027d89d Oct 30 19:32:40.526149 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:40.526055 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" event={"ID":"5ecee9ae-a839-4edc-b06b-cc014cb7e7ff","Type":"ContainerStarted","Data":"e328f93251d91e0deb76a6735b7f0e00b9dc38ab1732c2a819946ad63622431f"} Oct 30 19:32:40.526149 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:40.526111 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" event={"ID":"5ecee9ae-a839-4edc-b06b-cc014cb7e7ff","Type":"ContainerStarted","Data":"58f05fc19823e83257cf23e8664a5fff57fefdde0af8ab634f95cfb32027d89d"} Oct 30 19:32:40.526149 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:40.526140 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" Oct 30 19:32:40.611912 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:40.607963 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd"] Oct 30 19:32:40.611912 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:32:40.611491 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54b26b9_651b_4ba4_82a4_57819b4f02b4.slice/crio-ace17fb7a6c934f08695fda608d47e7ebc75392473d7dbfc237f4c935c37eb14 WatchSource:0}: Error finding container ace17fb7a6c934f08695fda608d47e7ebc75392473d7dbfc237f4c935c37eb14: Status 404 returned error can't find the container with id ace17fb7a6c934f08695fda608d47e7ebc75392473d7dbfc237f4c935c37eb14 Oct 30 19:32:40.640507 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:40.640463 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" podStartSLOduration=1.64044871 podStartE2EDuration="1.64044871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:40.639935891 +0000 UTC m=+1007.574723415" watchObservedRunningTime="2026-10-30 19:32:40.64044871 +0000 UTC m=+1007.575236232" Oct 30 19:32:40.804116 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:40.804025 4433 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 19:32:40.804390 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:40.804358 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" podUID="f2042f53-156d-42c2-9554-5fe559fbe56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 19:32:41.533527 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:41.533479 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" event={"ID":"f54b26b9-651b-4ba4-82a4-57819b4f02b4","Type":"ContainerStarted","Data":"aaddbbf3c4840d1108cad0187494b149c59a80b614ee09146440a08d8c5dc9d2"} Oct 30 19:32:41.533958 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:41.533546 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" event={"ID":"f54b26b9-651b-4ba4-82a4-57819b4f02b4","Type":"ContainerStarted","Data":"ace17fb7a6c934f08695fda608d47e7ebc75392473d7dbfc237f4c935c37eb14"} Oct 30 19:32:41.533958 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:41.533695 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" Oct 30 19:32:41.648346 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:41.648294 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" podStartSLOduration=2.6482812730000003 podStartE2EDuration="2.648281273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:41.646340389 +0000 UTC m=+1008.581127914" watchObservedRunningTime="2026-10-30 19:32:41.648281273 +0000 UTC m=+1008.583068797" Oct 30 19:32:41.802055 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:41.801968 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:41.802263 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:41.802241 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:41.802905 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:41.802887 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 19:32:44.005658 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:44.005630 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:44.244035 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:44.243989 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:44.454500 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:44.454470 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" Oct 30 19:32:44.490447 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:44.490412 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:46.269482 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:46.269447 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:46.888156 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:46.888124 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:47.175391 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:47.175310 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:49.435183 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:49.435147 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:49.661855 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:49.661824 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:49.894640 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:49.894602 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:51.537315 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:51.537289 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" Oct 30 19:32:52.542895 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:52.542866 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" Oct 30 19:32:53.366638 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:32:53.366602 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:32:54.708608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:54.708568 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:57.733389 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:57.733333 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:59.583882 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:32:59.583834 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:01.155636 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:01.143416 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:04.273906 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.273871 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s"] Oct 30 19:33:04.278933 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.278914 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:04.330762 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.330727 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-b5178f-serving-cert\"" Oct 30 19:33:04.370328 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.370295 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e2f76e-309b-46e6-8ee8-478c0def15fc-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:04.370328 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.370331 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmprz\" (UniqueName: \"kubernetes.io/projected/b0e2f76e-309b-46e6-8ee8-478c0def15fc-kube-api-access-gmprz\") pod \"maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:04.370547 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.370349 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-metrics-certs\") pod \"maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:04.370641 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.370617 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:04.382634 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.382609 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s"] Oct 30 19:33:04.471450 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.471409 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:04.471608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.471501 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e2f76e-309b-46e6-8ee8-478c0def15fc-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:04.471608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.471530 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmprz\" (UniqueName: \"kubernetes.io/projected/b0e2f76e-309b-46e6-8ee8-478c0def15fc-kube-api-access-gmprz\") pod \"maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:04.471608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.471555 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-metrics-certs\") pod \"maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:04.471608 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:33:04.471563 4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-serving-cert: secret "maas-api-e2e-worker-w4-b5178f-serving-cert" not found Oct 30 19:33:04.471862 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:33:04.471640 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-maas-api-tls podName:b0e2f76e-309b-46e6-8ee8-478c0def15fc nodeName:}" failed. No retries permitted until 2026-10-30 19:33:04.971618878 +0000 UTC m=+1031.906406408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-maas-api-tls") pod "maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" (UID: "b0e2f76e-309b-46e6-8ee8-478c0def15fc") : secret "maas-api-e2e-worker-w4-b5178f-serving-cert" not found Oct 30 19:33:04.472245 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.472220 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e2f76e-309b-46e6-8ee8-478c0def15fc-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:04.474054 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.474029 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-metrics-certs\") pod \"maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:04.532713 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.532630 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmprz\" (UniqueName: \"kubernetes.io/projected/b0e2f76e-309b-46e6-8ee8-478c0def15fc-kube-api-access-gmprz\") pod \"maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:04.551190 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.551140 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:04.778956 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.778868 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:04.974726 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.974691 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:04.978076 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:04.978052 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:05.009555 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:05.009510 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:05.209198 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:05.209166 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:05.580504 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:05.580469 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s"] Oct 30 19:33:05.582702 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:33:05.582675 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e2f76e_309b_46e6_8ee8_478c0def15fc.slice/crio-816c6104095d9b862c8ba6f7afdbffa2bc3e73bcb11107930904b553bd6c83f8 WatchSource:0}: Error finding container 816c6104095d9b862c8ba6f7afdbffa2bc3e73bcb11107930904b553bd6c83f8: Status 404 returned error can't find the container with id 816c6104095d9b862c8ba6f7afdbffa2bc3e73bcb11107930904b553bd6c83f8 Oct 30 19:33:05.666758 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:05.666722 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" event={"ID":"b0e2f76e-309b-46e6-8ee8-478c0def15fc","Type":"ContainerStarted","Data":"816c6104095d9b862c8ba6f7afdbffa2bc3e73bcb11107930904b553bd6c83f8"} Oct 30 19:33:06.673246 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:06.673213 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" event={"ID":"b0e2f76e-309b-46e6-8ee8-478c0def15fc","Type":"ContainerStarted","Data":"470e1bd3775727fcd5543a5bd559cacc7445ba02b4be589b9e6fe609a9fb2cb4"} Oct 30 19:33:06.673625 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:06.673286 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:06.798840 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:06.798739 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" podStartSLOduration=1.840485158 podStartE2EDuration="2.79872469s" totalImagesPullingTime="958.239532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:06.795940866 +0000 UTC m=+1033.730728391" watchObservedRunningTime="2026-10-30 19:33:06.79872469 +0000 UTC m=+1033.733512215" Oct 30 19:33:10.224097 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:10.224010 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:10.698398 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:10.698364 4433 generic.go:356] "Generic (PLEG): container finished" podID="3af9891c-2337-4531-a662-1eb43724b440" containerID="e65334778340ec7e024b969ec770140dc3783c3c5a2426d2cd1b1e42113c274f" exitCode=0 Oct 30 19:33:10.698398 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:10.698400 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" event={"ID":"3af9891c-2337-4531-a662-1eb43724b440","Type":"ContainerDied","Data":"e65334778340ec7e024b969ec770140dc3783c3c5a2426d2cd1b1e42113c274f"} Oct 30 19:33:10.698719 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:10.698704 4433 scope.go:122] "RemoveContainer" containerID="e65334778340ec7e024b969ec770140dc3783c3c5a2426d2cd1b1e42113c274f" Oct 30 19:33:10.699993 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:10.699954 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:33:10.700066 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:10.700048 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:33:10.700109 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:10.700079 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:33:11.716346 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:11.716311 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" event={"ID":"3af9891c-2337-4531-a662-1eb43724b440","Type":"ContainerStarted","Data":"4aa6244a56d4d2a5bbbafd45bb45bebfce9fd530567d772b12561b79ed53a8e8"} Oct 30 19:33:11.776033 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:11.776008 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:33:11.776238 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:11.776218 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:33:11.776298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:11.776273 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" podUID="3af9891c-2337-4531-a662-1eb43724b440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 19:33:11.910736 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:11.910644 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:12.657471 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:12.657438 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:12.681182 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:12.681147 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.59:8443/health\": dial tcp 10.134.0.59:8443: connect: connection refused" Oct 30 19:33:12.776547 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:12.776519 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:33:12.776944 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:12.776567 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" podUID="3af9891c-2337-4531-a662-1eb43724b440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 19:33:12.975871 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:12.975790 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:13.776920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:13.776883 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:33:13.777314 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:13.776938 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" podUID="3af9891c-2337-4531-a662-1eb43724b440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 19:33:14.776767 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:14.776731 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:33:14.776943 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:14.776783 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" podUID="3af9891c-2337-4531-a662-1eb43724b440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 19:33:15.409007 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:15.406446 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:15.776423 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:15.776341 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:33:15.776423 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:15.776395 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" podUID="3af9891c-2337-4531-a662-1eb43724b440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 19:33:16.749150 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:16.749122 4433 generic.go:356] "Generic (PLEG): container finished" podID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerID="470e1bd3775727fcd5543a5bd559cacc7445ba02b4be589b9e6fe609a9fb2cb4" exitCode=1 Oct 30 19:33:16.749512 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:16.749186 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" event={"ID":"b0e2f76e-309b-46e6-8ee8-478c0def15fc","Type":"ContainerDied","Data":"470e1bd3775727fcd5543a5bd559cacc7445ba02b4be589b9e6fe609a9fb2cb4"} Oct 30 19:33:16.749614 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:16.749596 4433 scope.go:122] "RemoveContainer" containerID="470e1bd3775727fcd5543a5bd559cacc7445ba02b4be589b9e6fe609a9fb2cb4" Oct 30 19:33:16.779620 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:16.779600 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:33:16.779945 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:16.779928 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:33:16.780546 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:16.780530 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 19:33:17.655017 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:17.654967 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:18.761234 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:18.761201 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" event={"ID":"b0e2f76e-309b-46e6-8ee8-478c0def15fc","Type":"ContainerStarted","Data":"237f94f087bccea7113b692d755fea5de7a65c09492ebd5a4483817982038331"} Oct 30 19:33:20.286656 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:20.286620 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:22.659233 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:22.659203 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:22.998440 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:22.998369 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:23.366115 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:33:23.366079 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:33:23.422157 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:23.422126 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:23.677121 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:23.677031 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:23.761532 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:23.761486 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:23.762883 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:23.762856 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.59:8443/health\": dial tcp 10.134.0.59:8443: connect: connection refused" Oct 30 19:33:23.763170 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:23.763138 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.59:8443/health\": dial tcp 10.134.0.59:8443: connect: connection refused" Oct 30 19:33:25.257716 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:25.257681 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:27.807727 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:27.807698 4433 generic.go:356] "Generic (PLEG): container finished" podID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerID="237f94f087bccea7113b692d755fea5de7a65c09492ebd5a4483817982038331" exitCode=1 Oct 30 19:33:27.808112 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:27.807735 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" event={"ID":"b0e2f76e-309b-46e6-8ee8-478c0def15fc","Type":"ContainerDied","Data":"237f94f087bccea7113b692d755fea5de7a65c09492ebd5a4483817982038331"} Oct 30 19:33:27.808112 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:27.807776 4433 scope.go:122] "RemoveContainer" containerID="470e1bd3775727fcd5543a5bd559cacc7445ba02b4be589b9e6fe609a9fb2cb4" Oct 30 19:33:27.808251 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:27.808133 4433 scope.go:122] "RemoveContainer" containerID="237f94f087bccea7113b692d755fea5de7a65c09492ebd5a4483817982038331" Oct 30 19:33:27.808390 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:33:27.808369 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s_odh-ai-gateway-infra(b0e2f76e-309b-46e6-8ee8-478c0def15fc)\"" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" Oct 30 19:33:29.384112 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:29.384076 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:30.456199 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:30.456168 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:32.668088 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:32.668053 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:33.142123 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:33.142090 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:33.475259 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:33.475174 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:35.212181 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:35.212141 4433 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:35.212690 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:35.212633 4433 scope.go:122] "RemoveContainer" containerID="237f94f087bccea7113b692d755fea5de7a65c09492ebd5a4483817982038331" Oct 30 19:33:35.212955 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:33:35.212931 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s_odh-ai-gateway-infra(b0e2f76e-309b-46e6-8ee8-478c0def15fc)\"" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" Oct 30 19:33:35.474396 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:35.474300 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:36.608586 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:36.608549 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:37.211420 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:37.211384 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:37.581153 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:37.581125 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:39.213005 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:39.212964 4433 scope.go:122] "RemoveContainer" containerID="237f94f087bccea7113b692d755fea5de7a65c09492ebd5a4483817982038331" Oct 30 19:33:40.405295 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:40.405262 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:40.886712 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:40.886682 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" event={"ID":"b0e2f76e-309b-46e6-8ee8-478c0def15fc","Type":"ContainerStarted","Data":"153743b6efcb900ca15ddae35e4c609df2a619e76e8fa8e2c3910f96cf9f6f0d"} Oct 30 19:33:40.886893 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:40.886878 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:45.089417 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:45.089383 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:46.892716 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:46.892676 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.59:8443/health\": dial tcp 10.134.0.59:8443: connect: connection refused" Oct 30 19:33:48.028614 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:48.028576 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:48.665018 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:48.664965 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:48.918139 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:48.918062 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:49.319606 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:49.319574 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:49.694480 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:49.694381 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:50.942019 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:50.941959 4433 generic.go:356] "Generic (PLEG): container finished" podID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerID="153743b6efcb900ca15ddae35e4c609df2a619e76e8fa8e2c3910f96cf9f6f0d" exitCode=1 Oct 30 19:33:50.942019 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:50.942007 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" event={"ID":"b0e2f76e-309b-46e6-8ee8-478c0def15fc","Type":"ContainerDied","Data":"153743b6efcb900ca15ddae35e4c609df2a619e76e8fa8e2c3910f96cf9f6f0d"} Oct 30 19:33:50.942551 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:50.942071 4433 scope.go:122] "RemoveContainer" containerID="237f94f087bccea7113b692d755fea5de7a65c09492ebd5a4483817982038331" Oct 30 19:33:50.942551 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:50.942477 4433 scope.go:122] "RemoveContainer" containerID="153743b6efcb900ca15ddae35e4c609df2a619e76e8fa8e2c3910f96cf9f6f0d" Oct 30 19:33:50.942757 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:33:50.942736 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s_odh-ai-gateway-infra(b0e2f76e-309b-46e6-8ee8-478c0def15fc)\"" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" Oct 30 19:33:52.586713 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.586673 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4"] Oct 30 19:33:52.591931 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.591908 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.697011 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.696957 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ada8d7\"/\"openshift-service-ca.crt\"" Oct 30 19:33:52.697011 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.696969 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ada8d7\"/\"kube-root-ca.crt\"" Oct 30 19:33:52.697253 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.697080 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ada8d7\"/\"default-dockercfg-7ntd4\"" Oct 30 19:33:52.697253 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.697143 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ada8d7\"/\"premium-w3-ada8d7-kserve-self-signed-certs\"" Oct 30 19:33:52.699402 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.699375 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4"] Oct 30 19:33:52.724517 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.724487 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-756rx\" (UniqueName: \"kubernetes.io/projected/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-kube-api-access-756rx\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.724674 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.724524 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-dshm\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.724674 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.724653 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-tmp-dir\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.724788 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.724720 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-model-cache\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.724832 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.724783 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-home\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.724870 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.724844 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-tls-certs\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.780692 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.780656 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:52.825892 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.825864 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-home\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.826066 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.825911 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-tls-certs\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.826066 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.825956 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-756rx\" (UniqueName: \"kubernetes.io/projected/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-kube-api-access-756rx\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.826066 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.825998 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-dshm\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.826066 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.826055 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-tmp-dir\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.826227 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.826100 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-model-cache\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.826352 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.826329 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-home\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.826478 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.826455 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-tmp-dir\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.826530 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.826485 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-model-cache\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.828174 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.828153 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-dshm\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.828415 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.828397 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-tls-certs\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.884098 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.884051 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-756rx\" (UniqueName: \"kubernetes.io/projected/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-kube-api-access-756rx\") pod \"premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:52.903456 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:52.903438 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:53.278112 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:53.278080 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4"] Oct 30 19:33:53.279466 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:33:53.279433 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2ffbe2b_c076_4ae7_994b_d8ab2db02185.slice/crio-094f3b6c5219b1bba85b73ab8290313b41b854adcf6fd44a8ce0e51cdac1af94 WatchSource:0}: Error finding container 094f3b6c5219b1bba85b73ab8290313b41b854adcf6fd44a8ce0e51cdac1af94: Status 404 returned error can't find the container with id 094f3b6c5219b1bba85b73ab8290313b41b854adcf6fd44a8ce0e51cdac1af94 Oct 30 19:33:53.365726 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:33:53.365696 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:33:53.963746 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:53.963716 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" event={"ID":"c2ffbe2b-c076-4ae7-994b-d8ab2db02185","Type":"ContainerStarted","Data":"0a28952560c20e4662b52d0e9ecd0bc4511843fb8313a36c88fcdc7dddc8fcb0"} Oct 30 19:33:53.964266 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:53.963758 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" event={"ID":"c2ffbe2b-c076-4ae7-994b-d8ab2db02185","Type":"ContainerStarted","Data":"094f3b6c5219b1bba85b73ab8290313b41b854adcf6fd44a8ce0e51cdac1af94"} Oct 30 19:33:54.080354 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:54.080277 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" podStartSLOduration=2.080263612 podStartE2EDuration="2.080263612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:54.079714481 +0000 UTC m=+1081.014502005" watchObservedRunningTime="2026-10-30 19:33:54.080263612 +0000 UTC m=+1081.015051136" Oct 30 19:33:54.622909 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:54.622877 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:54.904287 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:54.904222 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:33:55.212527 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:55.212455 4433 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:33:55.212891 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:55.212794 4433 scope.go:122] "RemoveContainer" containerID="153743b6efcb900ca15ddae35e4c609df2a619e76e8fa8e2c3910f96cf9f6f0d" Oct 30 19:33:55.213058 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:33:55.213037 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s_odh-ai-gateway-infra(b0e2f76e-309b-46e6-8ee8-478c0def15fc)\"" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" Oct 30 19:33:58.082361 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:58.082327 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:59.305306 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:59.305276 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:59.997689 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:33:59.997658 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:02.904054 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:02.904025 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:34:02.916296 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:02.916271 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:34:03.028462 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:03.028430 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:34:04.012723 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:04.012689 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:07.277519 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:07.277483 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:08.649875 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:08.649839 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:10.851872 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:10.851826 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:12.207158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:12.207130 4433 scope.go:122] "RemoveContainer" containerID="153743b6efcb900ca15ddae35e4c609df2a619e76e8fa8e2c3910f96cf9f6f0d" Oct 30 19:34:13.344483 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:13.344452 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:14.081410 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:14.081378 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" event={"ID":"b0e2f76e-309b-46e6-8ee8-478c0def15fc","Type":"ContainerStarted","Data":"aecbb010c96896016077a9a3ba4bd42784181015e31ca205b03b6d19fcbb5b2f"} Oct 30 19:34:14.081596 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:14.081578 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:34:14.104818 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:14.104791 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:14.956932 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:14.956896 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:17.181425 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.181392 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz"] Oct 30 19:34:17.185323 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.185305 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.277564 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.277533 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-model-cache\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.277721 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.277641 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-dshm\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.277721 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.277682 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-tls-certs\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.277721 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.277698 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vps9b\" (UniqueName: \"kubernetes.io/projected/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-kube-api-access-vps9b\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.277836 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.277720 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-tmp-dir\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.277899 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.277871 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-home\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.288700 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.288676 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-686689\"/\"simulator-w5-686689-kserve-self-signed-certs\"" Oct 30 19:34:17.288700 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.288695 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-686689\"/\"default-dockercfg-244mk\"" Oct 30 19:34:17.288831 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.288717 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-686689\"/\"openshift-service-ca.crt\"" Oct 30 19:34:17.288831 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.288742 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-686689\"/\"kube-root-ca.crt\"" Oct 30 19:34:17.291253 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.291223 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz"] Oct 30 19:34:17.378760 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.378727 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-model-cache\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.378957 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.378804 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-dshm\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.378957 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.378838 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-tls-certs\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.378957 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.378861 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vps9b\" (UniqueName: \"kubernetes.io/projected/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-kube-api-access-vps9b\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.378957 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.378890 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-tmp-dir\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.378957 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.378929 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-home\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.379315 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.379291 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-home\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.379437 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.379398 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-model-cache\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.379569 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.379548 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-tmp-dir\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.379677 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.379563 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:17.381333 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.381312 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-dshm\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.381571 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.381553 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-tls-certs\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.437230 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.437153 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vps9b\" (UniqueName: \"kubernetes.io/projected/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-kube-api-access-vps9b\") pod \"simulator-w5-686689-kserve-8667946bf5-r2pmz\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.497128 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.497104 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:17.870829 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.870803 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz"] Oct 30 19:34:17.872675 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:34:17.872643 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c6dccf0_37fd_41f6_ae17_ab1e44c13c89.slice/crio-aaf9caedc551b2e438798d5a4330dffa50847c04feb1104b8f424c0901a682e3 WatchSource:0}: Error finding container aaf9caedc551b2e438798d5a4330dffa50847c04feb1104b8f424c0901a682e3: Status 404 returned error can't find the container with id aaf9caedc551b2e438798d5a4330dffa50847c04feb1104b8f424c0901a682e3 Oct 30 19:34:17.984228 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:17.984195 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:18.103049 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:18.103010 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" event={"ID":"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89","Type":"ContainerStarted","Data":"6984a105a08251dc735276133323bedf9256f5fe7f63b92f1cee9dd1f06c0098"} Oct 30 19:34:18.103195 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:18.103058 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" event={"ID":"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89","Type":"ContainerStarted","Data":"aaf9caedc551b2e438798d5a4330dffa50847c04feb1104b8f424c0901a682e3"} Oct 30 19:34:18.219469 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:18.219361 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" podStartSLOduration=1.219342492 podStartE2EDuration="1.219342492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:18.219064691 +0000 UTC m=+1105.153852216" watchObservedRunningTime="2026-10-30 19:34:18.219342492 +0000 UTC m=+1105.154130016" Oct 30 19:34:18.497875 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:18.497805 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:20.091564 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:20.091539 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 19:34:22.854351 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:22.854316 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:23.365818 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:34:23.365776 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:34:27.497638 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:27.497607 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:27.510385 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:27.510363 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:27.542492 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:27.542466 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:27.998213 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:27.998181 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:28.167657 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:28.167626 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:34:28.471310 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:28.471261 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:29.085255 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:29.085208 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:29.326165 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:29.326116 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:32.546334 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:32.546304 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:37.204833 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:37.204764 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:41.717425 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:41.717394 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:41.978220 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:41.978133 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:46.637995 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:46.637946 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:48.277393 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:48.277305 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:51.260625 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:51.260583 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:53.364959 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:34:53.364922 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:34:56.063003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:56.062827 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:57.751237 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:57.751202 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:58.459157 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:34:58.459122 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:00.705435 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:00.705402 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:05.324498 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:05.324461 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:08.834271 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:08.834239 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c"] Oct 30 19:35:08.838346 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:08.838329 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:08.904878 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:08.904837 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cclq4\" (UniqueName: \"kubernetes.io/projected/0d290229-7e04-47a2-8c0a-c18988cd07f0-kube-api-access-cclq4\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:08.905052 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:08.904881 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-dshm\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:08.905052 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:08.905033 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-tmp-dir\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:08.905164 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:08.905065 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-home\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:08.905164 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:08.905093 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d290229-7e04-47a2-8c0a-c18988cd07f0-tls-certs\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:08.905273 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:08.905173 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-model-cache\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:08.939771 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:08.939732 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b5178f\"/\"openshift-service-ca.crt\"" Oct 30 19:35:08.939771 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:08.939735 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b5178f\"/\"kube-root-ca.crt\"" Oct 30 19:35:08.939949 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:08.939774 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b5178f\"/\"simulator-w4-b5178f-kserve-self-signed-certs\"" Oct 30 19:35:08.939949 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:08.939824 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b5178f\"/\"default-dockercfg-wljhm\"" Oct 30 19:35:08.942367 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:08.942349 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c"] Oct 30 19:35:09.005744 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:09.005711 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cclq4\" (UniqueName: \"kubernetes.io/projected/0d290229-7e04-47a2-8c0a-c18988cd07f0-kube-api-access-cclq4\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:09.005926 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:09.005750 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-dshm\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:09.005926 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:09.005814 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-tmp-dir\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:09.005926 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:09.005833 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-home\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:09.005926 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:09.005853 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d290229-7e04-47a2-8c0a-c18988cd07f0-tls-certs\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:09.006164 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:09.005910 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-model-cache\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:09.006358 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:09.006325 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-home\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:09.006358 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:09.006352 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-tmp-dir\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:09.006522 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:09.006390 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-model-cache\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:09.008027 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:09.008007 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-dshm\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:09.008371 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:09.008354 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d290229-7e04-47a2-8c0a-c18988cd07f0-tls-certs\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:09.014831 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:09.014803 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:09.062002 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:09.061961 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cclq4\" (UniqueName: \"kubernetes.io/projected/0d290229-7e04-47a2-8c0a-c18988cd07f0-kube-api-access-cclq4\") pod \"simulator-w4-b5178f-kserve-7559d7c57d-mxg5c\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:09.149360 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:09.149304 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:09.520023 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:09.519995 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c"] Oct 30 19:35:09.522114 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:35:09.522088 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d290229_7e04_47a2_8c0a_c18988cd07f0.slice/crio-07f51c8816638ab2d1b7f80dd4fc211a24da9c80c05fc082396c75e16a71ef25 WatchSource:0}: Error finding container 07f51c8816638ab2d1b7f80dd4fc211a24da9c80c05fc082396c75e16a71ef25: Status 404 returned error can't find the container with id 07f51c8816638ab2d1b7f80dd4fc211a24da9c80c05fc082396c75e16a71ef25 Oct 30 19:35:09.988690 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:09.988661 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:10.389017 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:10.388962 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" event={"ID":"0d290229-7e04-47a2-8c0a-c18988cd07f0","Type":"ContainerStarted","Data":"477cfe62a56919fc46f5a621408f63cc374fd884be57474c89a2a05cd8b287ac"} Oct 30 19:35:10.389173 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:10.389027 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" event={"ID":"0d290229-7e04-47a2-8c0a-c18988cd07f0","Type":"ContainerStarted","Data":"07f51c8816638ab2d1b7f80dd4fc211a24da9c80c05fc082396c75e16a71ef25"} Oct 30 19:35:10.503483 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:10.503436 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" podStartSLOduration=2.503422345 podStartE2EDuration="2.503422345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:10.50156732 +0000 UTC m=+1157.436354843" watchObservedRunningTime="2026-10-30 19:35:10.503422345 +0000 UTC m=+1157.438209870" Oct 30 19:35:11.150267 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:11.150237 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:14.594908 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:14.594876 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:19.149761 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:19.149726 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:19.161718 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:19.161695 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:19.373378 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:19.373331 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:19.451038 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:19.450956 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:35:23.364598 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:35:23.364561 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:35:24.028653 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:24.028618 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:26.103677 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:26.103645 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:28.334498 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:28.334471 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:28.703113 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:28.703030 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:30.519479 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:30.519444 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:33.312586 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:33.312554 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:38.215812 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.215780 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:38.442563 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.442531 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44"] Oct 30 19:35:38.446506 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.446488 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.547801 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.547729 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b5178f-openshift-default-dockercfg-k2spr\"" Oct 30 19:35:38.550572 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.550548 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44"] Oct 30 19:35:38.584035 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.584007 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.584163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.584059 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.584233 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.584180 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.584284 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.584233 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.584342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.584322 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.584378 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.584351 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7srjm\" (UniqueName: \"kubernetes.io/projected/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-kube-api-access-7srjm\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.584435 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.584376 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.584435 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.584400 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.584509 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.584492 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.685275 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.685250 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.685380 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.685279 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7srjm\" (UniqueName: \"kubernetes.io/projected/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-kube-api-access-7srjm\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.685380 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.685304 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.685380 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.685334 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.685380 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.685361 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.685591 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.685455 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.685591 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.685481 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.685591 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.685531 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.685591 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.685570 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.685859 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.685835 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.685931 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.685859 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.685931 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.685897 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.686156 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.686131 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.686276 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.686257 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.687610 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.687594 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.687823 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.687806 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.741850 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.741828 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.742248 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.742231 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7srjm\" (UniqueName: \"kubernetes.io/projected/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-kube-api-access-7srjm\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:38.760094 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:38.760072 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:39.183063 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:39.183041 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44"] Oct 30 19:35:39.184718 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:35:39.184693 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdd5ed0_278d_48ba_b7bc_b6bd929e23c6.slice/crio-07c1a069b9b99e32dfc2850733a1786aeba5a64a7e1aa9e74a2a96ab08e643fd WatchSource:0}: Error finding container 07c1a069b9b99e32dfc2850733a1786aeba5a64a7e1aa9e74a2a96ab08e643fd: Status 404 returned error can't find the container with id 07c1a069b9b99e32dfc2850733a1786aeba5a64a7e1aa9e74a2a96ab08e643fd Oct 30 19:35:39.187132 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:39.187103 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:35:39.187191 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:39.187167 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:35:39.187226 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:39.187197 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:35:39.548798 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:39.548714 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" event={"ID":"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6","Type":"ContainerStarted","Data":"81be2bc6c4e92f9a3900f58e6ec5162b23f65fc76b3c60bb22b1982248aefb6a"} Oct 30 19:35:39.548798 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:39.548771 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" event={"ID":"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6","Type":"ContainerStarted","Data":"07c1a069b9b99e32dfc2850733a1786aeba5a64a7e1aa9e74a2a96ab08e643fd"} Oct 30 19:35:39.666298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:39.666250 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" podStartSLOduration=1.6662357349999999 podStartE2EDuration="1.666235735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:39.664132073 +0000 UTC m=+1186.598919598" watchObservedRunningTime="2026-10-30 19:35:39.666235735 +0000 UTC m=+1186.601023258" Oct 30 19:35:39.760944 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:39.760917 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:40.762794 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:40.762761 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:35:40.763206 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:40.762813 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" podUID="fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:35:41.035993 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.035905 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb"] Oct 30 19:35:41.040722 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.040701 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.117431 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.117397 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb"] Oct 30 19:35:41.139359 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.139319 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-dshm\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.139359 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.139353 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eafb6a50-b7db-47d6-b742-86b92bfef67d-tls-certs\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.139591 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.139480 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr5qv\" (UniqueName: \"kubernetes.io/projected/eafb6a50-b7db-47d6-b742-86b92bfef67d-kube-api-access-hr5qv\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.139591 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.139575 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-home\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.139699 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.139624 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-tmp-dir\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.139699 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.139686 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-model-cache\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.159867 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.159842 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-686689\"/\"premium-w5-686689-kserve-self-signed-certs\"" Oct 30 19:35:41.216786 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.216708 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:41.241037 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.241007 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-model-cache\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.241193 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.241048 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-dshm\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.241193 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.241071 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eafb6a50-b7db-47d6-b742-86b92bfef67d-tls-certs\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.241193 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.241153 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr5qv\" (UniqueName: \"kubernetes.io/projected/eafb6a50-b7db-47d6-b742-86b92bfef67d-kube-api-access-hr5qv\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.241357 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.241237 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-home\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.241357 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.241309 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-tmp-dir\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.241499 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.241479 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-model-cache\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.241569 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.241556 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-home\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.241652 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.241635 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-tmp-dir\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.243206 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.243190 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-dshm\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.243574 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.243556 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eafb6a50-b7db-47d6-b742-86b92bfef67d-tls-certs\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.309800 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.309711 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr5qv\" (UniqueName: \"kubernetes.io/projected/eafb6a50-b7db-47d6-b742-86b92bfef67d-kube-api-access-hr5qv\") pod \"premium-w5-686689-kserve-5984d96bb6-v2zxb\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.355003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.354330 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:41.761272 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.761240 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:35:41.761435 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.761293 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" podUID="fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:35:41.767130 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:41.767109 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb"] Oct 30 19:35:41.768891 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:35:41.768866 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeafb6a50_b7db_47d6_b742_86b92bfef67d.slice/crio-65f620b0f6620cbe3de7432530eebf14c38aab1afda582e00b21438eff458e0b WatchSource:0}: Error finding container 65f620b0f6620cbe3de7432530eebf14c38aab1afda582e00b21438eff458e0b: Status 404 returned error can't find the container with id 65f620b0f6620cbe3de7432530eebf14c38aab1afda582e00b21438eff458e0b Oct 30 19:35:42.574933 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:42.574892 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" event={"ID":"eafb6a50-b7db-47d6-b742-86b92bfef67d","Type":"ContainerStarted","Data":"7c936addb1988b075839c20ebb39ef2a97566cec1425757ca505883408d161da"} Oct 30 19:35:42.574933 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:42.574940 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" event={"ID":"eafb6a50-b7db-47d6-b742-86b92bfef67d","Type":"ContainerStarted","Data":"65f620b0f6620cbe3de7432530eebf14c38aab1afda582e00b21438eff458e0b"} Oct 30 19:35:42.709001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:42.708925 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" podStartSLOduration=2.708908086 podStartE2EDuration="2.708908086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:42.707381223 +0000 UTC m=+1189.642168757" watchObservedRunningTime="2026-10-30 19:35:42.708908086 +0000 UTC m=+1189.643695612" Oct 30 19:35:42.760819 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:42.760786 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:35:42.760972 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:42.760869 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" podUID="fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:35:42.801318 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:42.801286 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:43.356133 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:43.356105 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:43.761334 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:43.761259 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:35:43.761334 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:43.761319 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" podUID="fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:35:44.760476 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:44.760440 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:35:44.760876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:44.760503 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" podUID="fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:35:45.764491 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:45.764461 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:45.764855 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:45.764718 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:45.765505 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:45.765490 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 19:35:47.445791 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:47.445759 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:51.356250 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:51.356222 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:51.368191 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:51.368169 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:51.637132 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:51.637058 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:35:52.072922 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:52.072889 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:53.367862 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:35:53.367828 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:35:56.743934 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:56.743885 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:58.785897 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:35:58.785866 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:01.393480 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:01.393448 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:06.034070 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:06.033995 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:10.756649 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:10.756620 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:15.377533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:15.377499 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:19.987723 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:19.987686 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:23.366351 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:36:23.366314 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:36:23.658653 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:23.658548 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:24.841940 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:24.841909 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:28.668576 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:28.668545 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:29.481421 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:29.481378 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:34.301950 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.301914 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:34.525730 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.525698 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj"] Oct 30 19:36:34.529661 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.529620 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.564740 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.564717 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.564839 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.564744 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.564839 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.564760 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.564915 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.564902 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.564958 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.564924 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5swxx\" (UniqueName: \"kubernetes.io/projected/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-kube-api-access-5swxx\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.565013 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.565002 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.565052 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.565024 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.565150 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.565138 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.565187 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.565162 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.582100 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.582078 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d52df0-openshift-default-dockercfg-dwj6l\"" Oct 30 19:36:34.635582 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.635550 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj"] Oct 30 19:36:34.665901 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.665876 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.666020 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.665907 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5swxx\" (UniqueName: \"kubernetes.io/projected/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-kube-api-access-5swxx\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.666020 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.665942 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.666020 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.665972 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.666194 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.666101 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.666194 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.666134 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.666299 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.666218 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.666299 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.666244 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.666299 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.666269 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.666467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.666444 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.666550 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.666523 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.666611 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.666542 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.666675 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.666658 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.666773 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.666754 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.668637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.668614 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.668758 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.668743 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.723504 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.723479 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.723639 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.723621 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5swxx\" (UniqueName: \"kubernetes.io/projected/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-kube-api-access-5swxx\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:34.842502 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:34.842452 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:35.274293 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:35.274268 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj"] Oct 30 19:36:35.275930 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:36:35.275906 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00c90c8_e436_49e8_aee6_e5f4c7a38cb4.slice/crio-38914cf7042f93a020c7c2b851a042ebdd6a107ca94647c8440c851718e7dc71 WatchSource:0}: Error finding container 38914cf7042f93a020c7c2b851a042ebdd6a107ca94647c8440c851718e7dc71: Status 404 returned error can't find the container with id 38914cf7042f93a020c7c2b851a042ebdd6a107ca94647c8440c851718e7dc71 Oct 30 19:36:35.278013 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:35.277970 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:36:35.278066 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:35.278045 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:36:35.278102 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:35.278072 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:36:35.874437 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:35.874398 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" event={"ID":"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4","Type":"ContainerStarted","Data":"4048e5777e14170ad9a2a0cbe7ce266ec9320a6aa26aac737f874db1ec5ee462"} Oct 30 19:36:35.874827 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:35.874456 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" event={"ID":"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4","Type":"ContainerStarted","Data":"38914cf7042f93a020c7c2b851a042ebdd6a107ca94647c8440c851718e7dc71"} Oct 30 19:36:36.103003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:36.095527 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" podStartSLOduration=2.095504728 podStartE2EDuration="2.095504728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:36.040441905 +0000 UTC m=+1242.975229427" watchObservedRunningTime="2026-10-30 19:36:36.095504728 +0000 UTC m=+1243.030292253" Oct 30 19:36:36.843283 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:36.843253 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:36.844940 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:36.844918 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:36:36.845062 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:36.844975 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" podUID="d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:36:37.847999 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:37.847344 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:36:37.847999 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:37.847420 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" podUID="d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:36:38.678543 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:38.678510 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:38.843124 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:38.843094 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:36:38.843288 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:38.843150 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" podUID="d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:36:38.948892 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:38.948805 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:39.843757 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:39.843720 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:36:39.843963 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:39.843784 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" podUID="d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:36:40.843195 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:40.843157 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:36:40.843578 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:40.843213 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" podUID="d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:36:41.846894 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:41.846854 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:41.847418 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:41.847308 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:41.847838 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:41.847821 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 19:36:43.712154 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:43.712121 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:48.594105 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:48.594063 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:53.215543 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:53.215514 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:53.366398 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:36:53.366359 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:36:58.091332 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:36:58.091301 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:02.757880 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:02.757845 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:05.761379 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:05.761340 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:07.386330 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:07.386298 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:12.057575 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:12.057543 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:16.761688 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:16.761651 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:21.423758 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:21.423719 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:23.366168 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:37:23.366137 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:37:26.064028 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:26.064000 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:30.848820 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:30.848785 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:34.937162 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:34.937079 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd"] Oct 30 19:37:34.942539 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:34.942518 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.046071 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.046045 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd"] Oct 30 19:37:35.063518 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.063490 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.063640 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.063570 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.063687 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.063669 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.063729 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.063707 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.063781 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.063736 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.063861 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.063839 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp2g2\" (UniqueName: \"kubernetes.io/projected/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-kube-api-access-pp2g2\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.063940 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.063922 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.064061 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.064040 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.064125 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.064091 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.165485 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.165462 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.165608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.165504 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.165608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.165530 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.165608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.165555 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.165608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.165578 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.165812 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.165612 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp2g2\" (UniqueName: \"kubernetes.io/projected/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-kube-api-access-pp2g2\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.165812 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.165668 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.165812 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.165720 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.165812 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.165744 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.166060 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.165945 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.166236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.166214 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.166347 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.166270 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.166417 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.166387 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.166544 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.166520 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.167854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.167834 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.168087 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.168071 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.221575 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.221523 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp2g2\" (UniqueName: \"kubernetes.io/projected/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-kube-api-access-pp2g2\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.221655 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.221604 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.256226 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.256191 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:35.481217 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.481111 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:35.685749 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.685726 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd"] Oct 30 19:37:35.687810 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:37:35.687781 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666f2ff7_bd9c_4525_bdf9_bda82db33b9a.slice/crio-dacd04ae61162905ac6d055bf05a763a1f3999860ee054341ee2eacf72c5f9eb WatchSource:0}: Error finding container dacd04ae61162905ac6d055bf05a763a1f3999860ee054341ee2eacf72c5f9eb: Status 404 returned error can't find the container with id dacd04ae61162905ac6d055bf05a763a1f3999860ee054341ee2eacf72c5f9eb Oct 30 19:37:35.690124 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.690092 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:35.690203 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.690157 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:35.690203 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:35.690185 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:36.263323 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:36.263288 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" event={"ID":"666f2ff7-bd9c-4525-bdf9-bda82db33b9a","Type":"ContainerStarted","Data":"5a957cc813899e54929c9ab214df6186723d60c904984bf84904ce57d5dee561"} Oct 30 19:37:36.263697 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:36.263342 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" event={"ID":"666f2ff7-bd9c-4525-bdf9-bda82db33b9a","Type":"ContainerStarted","Data":"dacd04ae61162905ac6d055bf05a763a1f3999860ee054341ee2eacf72c5f9eb"} Oct 30 19:37:36.379504 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:36.379455 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" podStartSLOduration=2.379440982 podStartE2EDuration="2.379440982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:36.377890676 +0000 UTC m=+1303.312678204" watchObservedRunningTime="2026-10-30 19:37:36.379440982 +0000 UTC m=+1303.314228571" Oct 30 19:37:37.258642 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:37.258610 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:37.260226 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:37.260197 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:37:37.260340 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:37.260246 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" podUID="666f2ff7-bd9c-4525-bdf9-bda82db33b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 19:37:38.258138 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:38.258104 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:37:38.258533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:38.258165 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" podUID="666f2ff7-bd9c-4525-bdf9-bda82db33b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 19:37:39.010254 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:39.010221 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:39.234953 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:39.234925 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:39.259073 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:39.259047 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:37:39.259423 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:39.259092 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" podUID="666f2ff7-bd9c-4525-bdf9-bda82db33b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 19:37:40.259069 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:40.259038 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:37:40.259447 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:40.259089 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" podUID="666f2ff7-bd9c-4525-bdf9-bda82db33b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 19:37:40.311734 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:40.311695 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:40.532490 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:40.532407 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:41.258859 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:41.258829 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:37:41.259035 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:41.258884 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" podUID="666f2ff7-bd9c-4525-bdf9-bda82db33b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 19:37:42.683304 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:42.683271 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:42.684031 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:42.683964 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:42.684800 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:42.684780 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:37:44.260215 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:44.260184 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c"] Oct 30 19:37:44.260752 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:44.260388 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" podUID="9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" containerName="istio-proxy" containerID="cri-o://3aecb66cd98f421e2448b3a39de52c44c9b98712bb2c7eb5eb8d17de972faa5f" gracePeriod=30 Oct 30 19:37:44.954661 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:44.954624 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:49.524531 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.524473 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:37:49.630674 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.630641 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-kube-api-access-mwmq7\" (UniqueName: \"kubernetes.io/projected/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-kube-api-access-mwmq7\") pod \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " Oct 30 19:37:49.630826 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.630687 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-envoy\") pod \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " Oct 30 19:37:49.630826 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.630744 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-credential-socket\") pod \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " Oct 30 19:37:49.630935 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.630844 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-data\") pod \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " Oct 30 19:37:49.630935 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.630874 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-podinfo\") pod \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " Oct 30 19:37:49.630935 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.630908 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istiod-ca-cert\") pod \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " Oct 30 19:37:49.631137 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.631006 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-workload-certs\") pod \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " Oct 30 19:37:49.631137 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.631078 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-token\" (UniqueName: \"kubernetes.io/projected/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-token\") pod \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " Oct 30 19:37:49.631137 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.631083 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-credential-socket" pod "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" (UID: "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:49.631284 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.631167 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-workload-socket\") pod \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\" (UID: \"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac\") " Oct 30 19:37:49.631410 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.631384 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-data" pod "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" (UID: "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:49.631575 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.631550 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-workload-certs" pod "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" (UID: "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:49.631637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.631557 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-workload-socket" pod "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" (UID: "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:49.631742 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.631719 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:49.631854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.631744 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:49.631854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.631760 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:49.631854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.631775 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:49.632017 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.631920 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istiod-ca-cert" pod "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" (UID: "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:49.633283 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.633259 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-kube-api-access-mwmq7" pod "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" (UID: "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac"). InnerVolumeSpecName "kube-api-access-mwmq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:49.633504 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.633480 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-envoy" pod "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" (UID: "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:49.633504 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.633497 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-token" pod "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" (UID: "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:49.633704 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.633681 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-podinfo" pod "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" (UID: "9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:49.732610 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.732574 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:49.732610 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.732601 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:49.732610 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.732615 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:49.732827 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.732626 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwmq7\" (UniqueName: \"kubernetes.io/projected/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-kube-api-access-mwmq7\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:49.732827 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.732635 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:49.758024 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:49.757996 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:50.365658 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:50.365625 4433 generic.go:356] "Generic (PLEG): container finished" podID="9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" containerID="3aecb66cd98f421e2448b3a39de52c44c9b98712bb2c7eb5eb8d17de972faa5f" exitCode=0 Oct 30 19:37:50.365855 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:50.365696 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" Oct 30 19:37:50.365855 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:50.365717 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" event={"ID":"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac","Type":"ContainerDied","Data":"3aecb66cd98f421e2448b3a39de52c44c9b98712bb2c7eb5eb8d17de972faa5f"} Oct 30 19:37:50.365855 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:50.365778 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c" event={"ID":"9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac","Type":"ContainerDied","Data":"21496e222dfa1f4342d4a2c01e983a1627012d3aa41cbf3a321f29f143360540"} Oct 30 19:37:50.365855 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:50.365805 4433 scope.go:122] "RemoveContainer" containerID="3aecb66cd98f421e2448b3a39de52c44c9b98712bb2c7eb5eb8d17de972faa5f" Oct 30 19:37:50.375326 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:50.375309 4433 scope.go:122] "RemoveContainer" containerID="3aecb66cd98f421e2448b3a39de52c44c9b98712bb2c7eb5eb8d17de972faa5f" Oct 30 19:37:50.375589 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:37:50.375569 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aecb66cd98f421e2448b3a39de52c44c9b98712bb2c7eb5eb8d17de972faa5f\": container with ID starting with 3aecb66cd98f421e2448b3a39de52c44c9b98712bb2c7eb5eb8d17de972faa5f not found: ID does not exist" containerID="3aecb66cd98f421e2448b3a39de52c44c9b98712bb2c7eb5eb8d17de972faa5f" Oct 30 19:37:50.375650 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:50.375601 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aecb66cd98f421e2448b3a39de52c44c9b98712bb2c7eb5eb8d17de972faa5f"} err="failed to get container status \"3aecb66cd98f421e2448b3a39de52c44c9b98712bb2c7eb5eb8d17de972faa5f\": rpc error: code = NotFound desc = could not find container \"3aecb66cd98f421e2448b3a39de52c44c9b98712bb2c7eb5eb8d17de972faa5f\": container with ID starting with 3aecb66cd98f421e2448b3a39de52c44c9b98712bb2c7eb5eb8d17de972faa5f not found: ID does not exist" Oct 30 19:37:50.579850 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:50.579812 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c"] Oct 30 19:37:50.586795 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:50.586771 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5n86c"] Oct 30 19:37:51.214966 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:51.214939 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" path="/var/lib/kubelet/pods/9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac/volumes" Oct 30 19:37:53.366309 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:37:53.366279 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:37:54.415269 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:54.415233 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:59.071948 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:59.071903 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:59.473012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:37:59.472885 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:01.954863 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:01.954832 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:02.180612 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:02.180579 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:03.714746 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:03.714712 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:04.130840 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:04.130809 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:08.461821 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:08.461793 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:13.056639 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:13.056600 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:17.731161 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:17.731126 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:22.329516 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:22.329478 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:23.366307 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:38:23.366266 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:38:27.160077 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:27.160044 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:28.909790 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:28.909755 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:29.143313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:29.143270 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:29.652308 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:29.652278 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4"] Oct 30 19:38:29.652484 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:29.652461 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" podUID="758501e2-8b8f-4465-93d3-6ba4ef2a033a" containerName="istio-proxy" containerID="cri-o://0e1222e4bf323bcf1651fb44038472726c1ff50c286c208bd360e1e89fe317d3" gracePeriod=30 Oct 30 19:38:31.831314 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:31.831282 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:34.722172 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:34.722140 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-9wgm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:38:34.722551 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:34.722234 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" podUID="758501e2-8b8f-4465-93d3-6ba4ef2a033a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:38:34.912534 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:34.912510 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:38:35.006253 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.006190 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-workload-socket\") pod \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " Oct 30 19:38:35.006253 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.006234 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758501e2-8b8f-4465-93d3-6ba4ef2a033a-kube-api-access-v8jbf\" (UniqueName: \"kubernetes.io/projected/758501e2-8b8f-4465-93d3-6ba4ef2a033a-kube-api-access-v8jbf\") pod \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " Oct 30 19:38:35.006441 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.006323 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-token\" (UniqueName: \"kubernetes.io/projected/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-token\") pod \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " Oct 30 19:38:35.006441 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.006420 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-workload-certs\") pod \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " Oct 30 19:38:35.006546 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.006450 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-workload-socket" pod "758501e2-8b8f-4465-93d3-6ba4ef2a033a" (UID: "758501e2-8b8f-4465-93d3-6ba4ef2a033a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:35.006546 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.006481 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-credential-socket\") pod \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " Oct 30 19:38:35.006546 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.006507 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-podinfo\") pod \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " Oct 30 19:38:35.006546 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.006541 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-data\") pod \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " Oct 30 19:38:35.006750 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.006600 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-envoy\") pod \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " Oct 30 19:38:35.006750 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.006630 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istiod-ca-cert\") pod \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\" (UID: \"758501e2-8b8f-4465-93d3-6ba4ef2a033a\") " Oct 30 19:38:35.006852 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.006805 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-workload-certs" pod "758501e2-8b8f-4465-93d3-6ba4ef2a033a" (UID: "758501e2-8b8f-4465-93d3-6ba4ef2a033a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:35.006942 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.006915 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-data" pod "758501e2-8b8f-4465-93d3-6ba4ef2a033a" (UID: "758501e2-8b8f-4465-93d3-6ba4ef2a033a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:35.007197 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.007145 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:35.007298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.007205 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:35.007298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.007220 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:35.007298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.007276 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-credential-socket" pod "758501e2-8b8f-4465-93d3-6ba4ef2a033a" (UID: "758501e2-8b8f-4465-93d3-6ba4ef2a033a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:35.007465 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.007381 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istiod-ca-cert" pod "758501e2-8b8f-4465-93d3-6ba4ef2a033a" (UID: "758501e2-8b8f-4465-93d3-6ba4ef2a033a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:35.009473 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.009437 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-podinfo" pod "758501e2-8b8f-4465-93d3-6ba4ef2a033a" (UID: "758501e2-8b8f-4465-93d3-6ba4ef2a033a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:38:35.009584 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.009504 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-envoy" pod "758501e2-8b8f-4465-93d3-6ba4ef2a033a" (UID: "758501e2-8b8f-4465-93d3-6ba4ef2a033a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:35.009796 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.009764 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758501e2-8b8f-4465-93d3-6ba4ef2a033a-kube-api-access-v8jbf" pod "758501e2-8b8f-4465-93d3-6ba4ef2a033a" (UID: "758501e2-8b8f-4465-93d3-6ba4ef2a033a"). InnerVolumeSpecName "kube-api-access-v8jbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:35.010250 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.010209 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-token" pod "758501e2-8b8f-4465-93d3-6ba4ef2a033a" (UID: "758501e2-8b8f-4465-93d3-6ba4ef2a033a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:35.108876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.108843 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8jbf\" (UniqueName: \"kubernetes.io/projected/758501e2-8b8f-4465-93d3-6ba4ef2a033a-kube-api-access-v8jbf\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:35.108876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.108872 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:35.108876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.108883 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:35.109120 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.108891 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:35.109120 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.108900 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:35.109120 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.108910 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758501e2-8b8f-4465-93d3-6ba4ef2a033a-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:35.679588 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.679561 4433 generic.go:356] "Generic (PLEG): container finished" podID="758501e2-8b8f-4465-93d3-6ba4ef2a033a" containerID="0e1222e4bf323bcf1651fb44038472726c1ff50c286c208bd360e1e89fe317d3" exitCode=0 Oct 30 19:38:35.679747 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.679599 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" event={"ID":"758501e2-8b8f-4465-93d3-6ba4ef2a033a","Type":"ContainerDied","Data":"0e1222e4bf323bcf1651fb44038472726c1ff50c286c208bd360e1e89fe317d3"} Oct 30 19:38:35.679747 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.679605 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" Oct 30 19:38:35.679747 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.679633 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4" event={"ID":"758501e2-8b8f-4465-93d3-6ba4ef2a033a","Type":"ContainerDied","Data":"545bf861220f0075e2f856cf3f581b28a2e0d335a65c03a8dfc63e56c90008ff"} Oct 30 19:38:35.679747 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.679652 4433 scope.go:122] "RemoveContainer" containerID="0e1222e4bf323bcf1651fb44038472726c1ff50c286c208bd360e1e89fe317d3" Oct 30 19:38:35.689089 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.689068 4433 scope.go:122] "RemoveContainer" containerID="0e1222e4bf323bcf1651fb44038472726c1ff50c286c208bd360e1e89fe317d3" Oct 30 19:38:35.689348 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:38:35.689328 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e1222e4bf323bcf1651fb44038472726c1ff50c286c208bd360e1e89fe317d3\": container with ID starting with 0e1222e4bf323bcf1651fb44038472726c1ff50c286c208bd360e1e89fe317d3 not found: ID does not exist" containerID="0e1222e4bf323bcf1651fb44038472726c1ff50c286c208bd360e1e89fe317d3" Oct 30 19:38:35.689423 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.689355 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e1222e4bf323bcf1651fb44038472726c1ff50c286c208bd360e1e89fe317d3"} err="failed to get container status \"0e1222e4bf323bcf1651fb44038472726c1ff50c286c208bd360e1e89fe317d3\": rpc error: code = NotFound desc = could not find container \"0e1222e4bf323bcf1651fb44038472726c1ff50c286c208bd360e1e89fe317d3\": container with ID starting with 0e1222e4bf323bcf1651fb44038472726c1ff50c286c208bd360e1e89fe317d3 not found: ID does not exist" Oct 30 19:38:35.892650 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.892581 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4"] Oct 30 19:38:35.941614 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:35.941564 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-9wgm4"] Oct 30 19:38:36.511972 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:36.511931 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:36.849235 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:36.849205 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:37.215649 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:37.215566 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758501e2-8b8f-4465-93d3-6ba4ef2a033a" path="/var/lib/kubelet/pods/758501e2-8b8f-4465-93d3-6ba4ef2a033a/volumes" Oct 30 19:38:41.321005 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:41.320952 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:42.499141 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:42.499107 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:46.059697 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:46.059640 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:49.384256 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:49.384223 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:50.953633 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:50.953598 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:52.866018 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:52.863011 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m"] Oct 30 19:38:52.866018 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:52.864298 4433 cpu_manager.go:414] "containerMap: removing container" podUID="9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" containerName="istio-proxy" Oct 30 19:38:52.866018 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:52.864320 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" containerName="istio-proxy" Oct 30 19:38:52.866018 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:52.864366 4433 cpu_manager.go:414] "containerMap: removing container" podUID="758501e2-8b8f-4465-93d3-6ba4ef2a033a" containerName="istio-proxy" Oct 30 19:38:52.866018 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:52.864373 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758501e2-8b8f-4465-93d3-6ba4ef2a033a" containerName="istio-proxy" Oct 30 19:38:52.866018 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:52.864661 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0046a0-bdf5-4975-a4e1-c40e43b7a7ac" containerName="istio-proxy" Oct 30 19:38:52.866018 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:52.864676 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="758501e2-8b8f-4465-93d3-6ba4ef2a033a" containerName="istio-proxy" Oct 30 19:38:52.869950 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:52.869928 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:52.923650 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:52.923612 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m"] Oct 30 19:38:52.969565 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:52.969540 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b5178f\"/\"distinct-w4-b5178f-kserve-self-signed-certs\"" Oct 30 19:38:52.979059 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:52.979036 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-model-cache\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:52.979155 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:52.979064 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c890add-9063-4067-bb5e-65324cf99b3d-tls-certs\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:52.979155 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:52.979146 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-home\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:52.979228 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:52.979160 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-dshm\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:52.979228 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:52.979176 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-tmp-dir\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:52.979228 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:52.979217 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvnnx\" (UniqueName: \"kubernetes.io/projected/6c890add-9063-4067-bb5e-65324cf99b3d-kube-api-access-nvnnx\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:53.029794 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.029758 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:53.080486 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.080454 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-model-cache\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:53.080659 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.080643 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c890add-9063-4067-bb5e-65324cf99b3d-tls-certs\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:53.080741 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.080729 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-home\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:53.080798 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.080762 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-dshm\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:53.080798 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.080778 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-tmp-dir\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:53.080798 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.080795 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvnnx\" (UniqueName: \"kubernetes.io/projected/6c890add-9063-4067-bb5e-65324cf99b3d-kube-api-access-nvnnx\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:53.080959 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.080858 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-model-cache\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:53.081037 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.080975 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-home\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:53.081163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.081145 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-tmp-dir\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:53.082911 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.082889 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-dshm\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:53.083214 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.083196 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c890add-9063-4067-bb5e-65324cf99b3d-tls-certs\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:53.136647 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.136590 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvnnx\" (UniqueName: \"kubernetes.io/projected/6c890add-9063-4067-bb5e-65324cf99b3d-kube-api-access-nvnnx\") pod \"distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:53.182652 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.182629 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:53.365429 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:38:53.365400 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:38:53.572586 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.572564 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m"] Oct 30 19:38:53.574093 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:38:53.574058 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c890add_9063_4067_bb5e_65324cf99b3d.slice/crio-d030cd067d51e9b83df9dc93ffd9499e7176c2338c8d3da10ce3ffb56f9ad117 WatchSource:0}: Error finding container d030cd067d51e9b83df9dc93ffd9499e7176c2338c8d3da10ce3ffb56f9ad117: Status 404 returned error can't find the container with id d030cd067d51e9b83df9dc93ffd9499e7176c2338c8d3da10ce3ffb56f9ad117 Oct 30 19:38:53.780756 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.780715 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" event={"ID":"6c890add-9063-4067-bb5e-65324cf99b3d","Type":"ContainerStarted","Data":"1875ecf370b68a6a735ce86d489ffd829a18b283b23b706b92d6773b3e75b123"} Oct 30 19:38:53.780756 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.780762 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" event={"ID":"6c890add-9063-4067-bb5e-65324cf99b3d","Type":"ContainerStarted","Data":"d030cd067d51e9b83df9dc93ffd9499e7176c2338c8d3da10ce3ffb56f9ad117"} Oct 30 19:38:53.894928 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:53.894827 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" podStartSLOduration=1.894810755 podStartE2EDuration="1.894810755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:53.89317669 +0000 UTC m=+1380.827964213" watchObservedRunningTime="2026-10-30 19:38:53.894810755 +0000 UTC m=+1380.829598280" Oct 30 19:38:54.183296 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:54.183217 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:38:55.712339 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:55.712308 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:58.342123 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:38:58.342089 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:00.317940 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:00.317906 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:03.183618 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:03.183587 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:39:03.195772 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:03.195747 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:39:03.847351 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:03.847320 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:39:04.472742 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:04.472710 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:05.091341 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:05.091307 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:09.740403 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:09.740343 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:10.773006 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.772946 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7"] Oct 30 19:39:10.776920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.776900 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.868436 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.868397 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-dshm\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.868603 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.868446 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-home\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.868603 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.868518 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-model-cache\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.868728 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.868662 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f1295b7-8682-4eed-b904-83b8b6537e8b-tls-certs\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.868804 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.868784 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8dzq\" (UniqueName: \"kubernetes.io/projected/7f1295b7-8682-4eed-b904-83b8b6537e8b-kube-api-access-v8dzq\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.868860 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.868848 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-tmp-dir\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.881080 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.881047 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a3f9b7\"/\"openshift-service-ca.crt\"" Oct 30 19:39:10.881254 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.881229 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a3f9b7\"/\"default-dockercfg-x47rt\"" Oct 30 19:39:10.881341 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.881229 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a3f9b7\"/\"kube-root-ca.crt\"" Oct 30 19:39:10.881341 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.881312 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a3f9b7\"/\"distinct-2-w2-a3f9b7-kserve-self-signed-certs\"" Oct 30 19:39:10.885430 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.885406 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7"] Oct 30 19:39:10.945698 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.945650 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:10.969657 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.969631 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8dzq\" (UniqueName: \"kubernetes.io/projected/7f1295b7-8682-4eed-b904-83b8b6537e8b-kube-api-access-v8dzq\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.969809 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.969677 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-tmp-dir\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.969809 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.969723 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-dshm\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.969809 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.969751 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-home\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.969809 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.969783 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-model-cache\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.970067 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.969842 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f1295b7-8682-4eed-b904-83b8b6537e8b-tls-certs\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.970205 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.970181 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-tmp-dir\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.970205 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.970196 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-home\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.970291 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.970222 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-model-cache\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.972209 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.972186 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-dshm\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:10.972357 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:10.972342 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f1295b7-8682-4eed-b904-83b8b6537e8b-tls-certs\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:11.028379 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:11.028326 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8dzq\" (UniqueName: \"kubernetes.io/projected/7f1295b7-8682-4eed-b904-83b8b6537e8b-kube-api-access-v8dzq\") pod \"distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:11.089012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:11.088962 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:11.465633 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:11.465601 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7"] Oct 30 19:39:11.467344 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:39:11.467308 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f1295b7_8682_4eed_b904_83b8b6537e8b.slice/crio-e88ecb19b9aff58f14ecda10607c60f3873edf48e1c001c144ee5ef5ffa1e114 WatchSource:0}: Error finding container e88ecb19b9aff58f14ecda10607c60f3873edf48e1c001c144ee5ef5ffa1e114: Status 404 returned error can't find the container with id e88ecb19b9aff58f14ecda10607c60f3873edf48e1c001c144ee5ef5ffa1e114 Oct 30 19:39:11.886173 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:11.886137 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" event={"ID":"7f1295b7-8682-4eed-b904-83b8b6537e8b","Type":"ContainerStarted","Data":"90fc945efa9e5b9398d37a47ac70c5348a2bc0b570869f6a5b3fbf618fc6f38b"} Oct 30 19:39:11.886543 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:11.886180 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" event={"ID":"7f1295b7-8682-4eed-b904-83b8b6537e8b","Type":"ContainerStarted","Data":"e88ecb19b9aff58f14ecda10607c60f3873edf48e1c001c144ee5ef5ffa1e114"} Oct 30 19:39:12.001477 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:12.001431 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" podStartSLOduration=2.00141742 podStartE2EDuration="2.00141742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:12.000291342 +0000 UTC m=+1398.935078895" watchObservedRunningTime="2026-10-30 19:39:12.00141742 +0000 UTC m=+1398.936204943" Oct 30 19:39:12.089911 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:12.089887 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:14.375694 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:14.375641 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:19.012415 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:19.012384 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:21.089379 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:21.089349 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:21.101208 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:21.101177 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:21.951673 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:21.951641 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:39:23.366498 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:39:23.366464 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:39:23.650567 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:23.650453 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:25.182332 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.182300 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr"] Oct 30 19:39:25.186950 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.186925 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.289629 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.289598 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr"] Oct 30 19:39:25.303361 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.303327 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.303519 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.303395 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.303583 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.303534 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.303645 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.303602 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.303785 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.303763 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2583f5a9-8bc2-45f0-9470-418c96a00d14-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.303858 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.303814 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.303858 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.303843 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hgkm\" (UniqueName: \"kubernetes.io/projected/2583f5a9-8bc2-45f0-9470-418c96a00d14-kube-api-access-4hgkm\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.303963 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.303890 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.304055 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.304012 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.405327 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.405303 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.405447 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.405353 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.405447 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.405381 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.405447 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.405420 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2583f5a9-8bc2-45f0-9470-418c96a00d14-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.405565 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.405457 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.405631 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.405596 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hgkm\" (UniqueName: \"kubernetes.io/projected/2583f5a9-8bc2-45f0-9470-418c96a00d14-kube-api-access-4hgkm\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.405693 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.405652 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.405746 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.405693 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.405799 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.405749 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.405865 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.405810 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.405865 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.405827 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.406107 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.406082 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.406230 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.406134 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2583f5a9-8bc2-45f0-9470-418c96a00d14-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.406230 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.406171 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.407737 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.407718 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.407836 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.407818 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.461892 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.461841 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.461992 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.461964 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hgkm\" (UniqueName: \"kubernetes.io/projected/2583f5a9-8bc2-45f0-9470-418c96a00d14-kube-api-access-4hgkm\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.502279 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.502261 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:25.930232 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.930200 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr"] Oct 30 19:39:25.935099 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:39:25.935060 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2583f5a9_8bc2_45f0_9470_418c96a00d14.slice/crio-b5b8850d687a2d616a2a0c18fd81811b3e398a2fde4d69ff502a1ee689ec5210 WatchSource:0}: Error finding container b5b8850d687a2d616a2a0c18fd81811b3e398a2fde4d69ff502a1ee689ec5210: Status 404 returned error can't find the container with id b5b8850d687a2d616a2a0c18fd81811b3e398a2fde4d69ff502a1ee689ec5210 Oct 30 19:39:25.938137 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.938098 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:25.938232 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.938191 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:25.938290 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.938237 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:25.969394 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:25.969232 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" event={"ID":"2583f5a9-8bc2-45f0-9470-418c96a00d14","Type":"ContainerStarted","Data":"b5b8850d687a2d616a2a0c18fd81811b3e398a2fde4d69ff502a1ee689ec5210"} Oct 30 19:39:26.977640 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:26.977604 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" event={"ID":"2583f5a9-8bc2-45f0-9470-418c96a00d14","Type":"ContainerStarted","Data":"eaa19f3fbc87685d8e96fe26d9edab6acbf2ec034e11a43cdcf7db6b9881b9ed"} Oct 30 19:39:27.147033 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:27.146966 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" podStartSLOduration=2.146952931 podStartE2EDuration="2.146952931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:27.144164427 +0000 UTC m=+1414.078951950" watchObservedRunningTime="2026-10-30 19:39:27.146952931 +0000 UTC m=+1414.081740454" Oct 30 19:39:27.509534 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:27.509497 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:27.511111 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:27.510721 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:39:27.511111 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:27.510827 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" podUID="2583f5a9-8bc2-45f0-9470-418c96a00d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:39:27.707867 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:27.707831 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:28.509003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:28.508911 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:39:28.509003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:28.508969 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" podUID="2583f5a9-8bc2-45f0-9470-418c96a00d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:39:28.577629 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:28.577598 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:29.509345 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:29.509307 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:39:29.509825 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:29.509377 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" podUID="2583f5a9-8bc2-45f0-9470-418c96a00d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:39:30.505250 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:30.505215 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:39:30.505529 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:30.505496 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" podUID="2583f5a9-8bc2-45f0-9470-418c96a00d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:39:31.507060 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:31.507025 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:39:31.507608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:31.507578 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" podUID="2583f5a9-8bc2-45f0-9470-418c96a00d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:39:32.503274 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:32.503234 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:39:32.503476 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:32.503293 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" podUID="2583f5a9-8bc2-45f0-9470-418c96a00d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:39:33.391250 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:33.391209 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:33.507268 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:33.507239 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:33.507629 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:33.507607 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:33.508390 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:33.508371 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:39:35.869248 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:35.869200 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz"] Oct 30 19:39:35.873136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:35.873114 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:35.921477 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:35.921453 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:35.921608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:35.921490 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:35.921679 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:35.921597 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:35.921679 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:35.921662 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:35.921791 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:35.921696 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:35.921791 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:35.921730 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:35.921893 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:35.921841 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv6sj\" (UniqueName: \"kubernetes.io/projected/4ca58922-83fb-40bb-a289-1fcd9114ffd5-kube-api-access-lv6sj\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:35.921949 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:35.921892 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:35.921949 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:35.921920 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:35.976458 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:35.976427 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz"] Oct 30 19:39:36.022683 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.022658 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.022829 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.022690 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.022829 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.022709 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.022829 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.022734 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.022829 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.022789 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv6sj\" (UniqueName: \"kubernetes.io/projected/4ca58922-83fb-40bb-a289-1fcd9114ffd5-kube-api-access-lv6sj\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.022829 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.022813 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.023136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.022836 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.023136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.022883 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.023245 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.023203 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.023301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.023245 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.023355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.023298 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.023416 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.023369 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.023559 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.023540 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.023633 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.023578 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.025304 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.025281 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.025389 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.025344 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.079443 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.079419 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.079546 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.079483 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv6sj\" (UniqueName: \"kubernetes.io/projected/4ca58922-83fb-40bb-a289-1fcd9114ffd5-kube-api-access-lv6sj\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.187093 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.187037 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:36.611907 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.611878 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz"] Oct 30 19:39:36.613708 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:39:36.613675 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca58922_83fb_40bb_a289_1fcd9114ffd5.slice/crio-727eef303c1dcc60d07d75f546821b3267d5dc81c3b23f4d248ba166a83a9db6 WatchSource:0}: Error finding container 727eef303c1dcc60d07d75f546821b3267d5dc81c3b23f4d248ba166a83a9db6: Status 404 returned error can't find the container with id 727eef303c1dcc60d07d75f546821b3267d5dc81c3b23f4d248ba166a83a9db6 Oct 30 19:39:36.616124 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.616091 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:36.616226 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.616155 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:36.616226 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:36.616184 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:37.056517 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:37.056422 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" event={"ID":"4ca58922-83fb-40bb-a289-1fcd9114ffd5","Type":"ContainerStarted","Data":"388429f376318787e07d067cef219db9385b1ae7a93259ad54629263c9b254e7"} Oct 30 19:39:37.056517 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:37.056479 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" event={"ID":"4ca58922-83fb-40bb-a289-1fcd9114ffd5","Type":"ContainerStarted","Data":"727eef303c1dcc60d07d75f546821b3267d5dc81c3b23f4d248ba166a83a9db6"} Oct 30 19:39:37.175408 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:37.175360 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" podStartSLOduration=2.1753467459999998 podStartE2EDuration="2.175346746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:37.173665254 +0000 UTC m=+1424.108452779" watchObservedRunningTime="2026-10-30 19:39:37.175346746 +0000 UTC m=+1424.110134269" Oct 30 19:39:37.205095 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:37.205050 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:37.208267 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:37.208236 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:39:37.208481 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:37.208456 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" podUID="4ca58922-83fb-40bb-a289-1fcd9114ffd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:39:38.011477 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:38.011438 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:38.203004 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:38.188284 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:39:38.203004 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:38.188351 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" podUID="4ca58922-83fb-40bb-a289-1fcd9114ffd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:39:39.205298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:39.205261 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:39:39.205868 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:39.205836 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" podUID="4ca58922-83fb-40bb-a289-1fcd9114ffd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:39:40.207002 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:40.206301 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:39:40.207002 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:40.206369 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" podUID="4ca58922-83fb-40bb-a289-1fcd9114ffd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:39:41.202004 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:41.194260 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:39:41.202004 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:41.194330 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" podUID="4ca58922-83fb-40bb-a289-1fcd9114ffd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:39:42.206356 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:42.206319 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:39:42.206897 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:42.206865 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" podUID="4ca58922-83fb-40bb-a289-1fcd9114ffd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:39:42.687612 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:42.687579 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:43.187594 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:43.187558 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:39:43.187757 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:43.187624 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" podUID="4ca58922-83fb-40bb-a289-1fcd9114ffd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:39:44.192074 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:44.192044 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:44.192497 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:44.192458 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:44.193166 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:44.193150 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:39:47.311125 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:47.311089 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:49.733245 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:49.733209 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:52.063153 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:52.063117 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:53.365886 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:39:53.365845 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:39:56.660429 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:39:56.660399 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:00.420163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.420130 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v"] Oct 30 19:40:00.424202 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.424185 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.484893 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.484873 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-a3f9b7-openshift-default-dockercfg-9v6f2\"" Oct 30 19:40:00.492586 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.492560 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.492704 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.492590 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.492704 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.492622 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.492818 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.492726 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.492818 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.492773 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.492895 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.492869 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.492952 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.492940 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2d4k\" (UniqueName: \"kubernetes.io/projected/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-kube-api-access-v2d4k\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.493045 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.493026 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.493094 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.493058 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.540985 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.540959 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v"] Oct 30 19:40:00.593932 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.593905 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.594072 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.593939 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.594072 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.593965 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.594072 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.594020 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.594072 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.594055 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.594271 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.594097 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.594271 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.594163 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2d4k\" (UniqueName: \"kubernetes.io/projected/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-kube-api-access-v2d4k\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.594271 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.594224 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.594271 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.594260 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.594517 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.594493 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.594577 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.594533 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.594672 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.594648 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.594672 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.594666 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.594836 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.594813 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.596210 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.596192 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.596430 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.596411 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.652269 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.652242 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.652398 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.652382 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2d4k\" (UniqueName: \"kubernetes.io/projected/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-kube-api-access-v2d4k\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:00.737396 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:00.737343 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:01.162136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:01.162107 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v"] Oct 30 19:40:01.164136 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:40:01.164111 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c1ea2e_e33e_4828_abd3_266fcc1a8dbb.slice/crio-fe898419564f6e7ccac4e7af72f5cdc3510d55b2aa41183e60904afdee2d7cc5 WatchSource:0}: Error finding container fe898419564f6e7ccac4e7af72f5cdc3510d55b2aa41183e60904afdee2d7cc5: Status 404 returned error can't find the container with id fe898419564f6e7ccac4e7af72f5cdc3510d55b2aa41183e60904afdee2d7cc5 Oct 30 19:40:01.166334 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:01.166301 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:40:01.166424 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:01.166367 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:40:01.166424 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:01.166396 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:40:01.197187 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:01.197156 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" event={"ID":"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb","Type":"ContainerStarted","Data":"fe898419564f6e7ccac4e7af72f5cdc3510d55b2aa41183e60904afdee2d7cc5"} Oct 30 19:40:01.470307 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:01.470233 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:02.217355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:02.217317 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" event={"ID":"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb","Type":"ContainerStarted","Data":"0ee053c921b54468e37d4194dd846c5ef828ddd9a783d6248cfc6d5fc984b7d1"} Oct 30 19:40:02.409499 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:02.409431 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" podStartSLOduration=2.40941337 podStartE2EDuration="2.40941337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:02.402476589 +0000 UTC m=+1449.337264113" watchObservedRunningTime="2026-10-30 19:40:02.40941337 +0000 UTC m=+1449.344200895" Oct 30 19:40:02.738314 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:02.738283 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:02.739912 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:02.739891 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:40:02.740045 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:02.739943 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" podUID="75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:40:03.738270 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:03.738235 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:40:03.738436 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:03.738297 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" podUID="75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:40:04.738502 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:04.738467 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:40:04.738900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:04.738529 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" podUID="75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:40:04.808910 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:04.808811 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:05.737971 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:05.737929 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:40:05.738174 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:05.738036 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" podUID="75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:40:06.310673 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:06.310638 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:06.737847 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:06.737817 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:40:06.738035 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:06.737864 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" podUID="75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:40:07.737987 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:07.737951 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:40:07.738363 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:07.738025 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" podUID="75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:40:08.742223 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:08.742195 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:08.742597 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:08.742562 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:08.743155 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:08.743137 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 19:40:11.071083 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:11.071041 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:11.328339 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:11.328248 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:15.765310 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:15.765275 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:20.556636 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:20.556569 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:23.365335 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:40:23.365294 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:40:25.185203 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:25.185166 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:29.894461 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:29.894400 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:34.471416 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:34.471379 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:37.374401 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.374361 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j"] Oct 30 19:40:37.378479 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.378457 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.436753 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.436713 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j"] Oct 30 19:40:37.475105 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.475078 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65fpl\" (UniqueName: \"kubernetes.io/projected/7c9c64c5-157e-4750-9a42-3c2b7f15e741-kube-api-access-65fpl\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.475268 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.475122 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-model-cache\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.475268 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.475211 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c9c64c5-157e-4750-9a42-3c2b7f15e741-tls-certs\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.475372 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.475327 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-tmp-dir\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.475449 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.475433 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-dshm\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.475510 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.475499 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-home\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.482478 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.482455 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f55cd\"/\"distinct-w6-5f55cd-kserve-self-signed-certs\"" Oct 30 19:40:37.482611 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.482598 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5f55cd\"/\"openshift-service-ca.crt\"" Oct 30 19:40:37.482670 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.482632 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f55cd\"/\"default-dockercfg-z7q6k\"" Oct 30 19:40:37.482814 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.482798 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5f55cd\"/\"kube-root-ca.crt\"" Oct 30 19:40:37.564704 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.564663 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:37.576453 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.576425 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-dshm\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.576577 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.576471 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-home\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.576577 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.576512 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65fpl\" (UniqueName: \"kubernetes.io/projected/7c9c64c5-157e-4750-9a42-3c2b7f15e741-kube-api-access-65fpl\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.576577 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.576534 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-model-cache\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.576577 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.576557 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c9c64c5-157e-4750-9a42-3c2b7f15e741-tls-certs\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.576774 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.576629 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-tmp-dir\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.576997 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.576917 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-home\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.576997 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.576962 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-model-cache\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.577109 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.577078 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-tmp-dir\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.578613 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.578597 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-dshm\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.579067 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.579047 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c9c64c5-157e-4750-9a42-3c2b7f15e741-tls-certs\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.633056 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.632994 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65fpl\" (UniqueName: \"kubernetes.io/projected/7c9c64c5-157e-4750-9a42-3c2b7f15e741-kube-api-access-65fpl\") pod \"distinct-w6-5f55cd-kserve-599869cd77-l878j\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:37.692167 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:37.692143 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:38.074180 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:38.073154 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j"] Oct 30 19:40:38.077177 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:40:38.077151 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c9c64c5_157e_4750_9a42_3c2b7f15e741.slice/crio-5ddf4385aa5381570f583a74fc486ce28ef3e7704f1389d1881e3fd175451c4d WatchSource:0}: Error finding container 5ddf4385aa5381570f583a74fc486ce28ef3e7704f1389d1881e3fd175451c4d: Status 404 returned error can't find the container with id 5ddf4385aa5381570f583a74fc486ce28ef3e7704f1389d1881e3fd175451c4d Oct 30 19:40:38.421661 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:38.421573 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" event={"ID":"7c9c64c5-157e-4750-9a42-3c2b7f15e741","Type":"ContainerStarted","Data":"746f076b7eede5cf82e4a802bfdfc3e92a3befbab05753a6218103ce8c8d7a69"} Oct 30 19:40:38.421661 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:38.421615 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" event={"ID":"7c9c64c5-157e-4750-9a42-3c2b7f15e741","Type":"ContainerStarted","Data":"5ddf4385aa5381570f583a74fc486ce28ef3e7704f1389d1881e3fd175451c4d"} Oct 30 19:40:38.541039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:38.540973 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" podStartSLOduration=1.540960361 podStartE2EDuration="1.540960361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:38.538822834 +0000 UTC m=+1485.473610358" watchObservedRunningTime="2026-10-30 19:40:38.540960361 +0000 UTC m=+1485.475747915" Oct 30 19:40:38.693048 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:38.692902 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:39.283199 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:39.283167 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:44.022176 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:44.022135 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:45.233661 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:45.233616 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:47.699311 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:47.699279 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:47.718171 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:47.718141 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:48.504348 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:48.504315 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:40:48.747330 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:48.747287 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:50.065292 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:50.065251 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:53.370892 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:40:53.370851 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:40:53.429906 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:53.429878 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:58.111294 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:40:58.111263 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:00.085709 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:00.085663 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:01.790431 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:01.790399 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:02.853061 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:02.853031 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:05.138176 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:05.138143 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:07.563579 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:07.563547 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:12.489439 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:12.489385 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:16.128025 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.127975 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm"] Oct 30 19:41:16.131960 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.131940 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.183354 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.183323 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2d9861-openshift-default-dockercfg-ngs2l\"" Oct 30 19:41:16.236056 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.236022 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm"] Oct 30 19:41:16.272689 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.272650 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.272853 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.272798 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.272853 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.272839 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb7ln\" (UniqueName: \"kubernetes.io/projected/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-kube-api-access-sb7ln\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.273021 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.272971 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.273107 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.273088 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.273200 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.273185 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.273245 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.273221 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.273287 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.273253 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.273336 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.273308 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.373993 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.373943 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.374163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.374032 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.374163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.374055 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.374163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.374076 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.374163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.374093 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.374384 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.374165 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.374384 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.374341 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.374494 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.374391 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.374494 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.374403 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb7ln\" (UniqueName: \"kubernetes.io/projected/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-kube-api-access-sb7ln\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.374494 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.374459 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.374494 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.374464 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.374743 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.374720 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.374817 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.374766 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.374870 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.374813 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.376309 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.376286 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.376555 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.376535 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.431591 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.431509 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.431832 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.431812 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb7ln\" (UniqueName: \"kubernetes.io/projected/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-kube-api-access-sb7ln\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.445058 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.445026 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:16.875090 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.875066 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm"] Oct 30 19:41:16.876118 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:41:16.876088 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe0d869_8b60_4ef3_acf4_8a0c2375b35c.slice/crio-06e6623acfed759fd5eba22bf5e31d175d568452797fd5f4f66ac414db8bfac4 WatchSource:0}: Error finding container 06e6623acfed759fd5eba22bf5e31d175d568452797fd5f4f66ac414db8bfac4: Status 404 returned error can't find the container with id 06e6623acfed759fd5eba22bf5e31d175d568452797fd5f4f66ac414db8bfac4 Oct 30 19:41:16.878506 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.878471 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:41:16.878594 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.878538 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:41:16.878594 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:16.878570 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:41:17.226857 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.226779 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k"] Oct 30 19:41:17.230631 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.230609 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.284422 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.284389 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-tmp-dir\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.284422 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.284427 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-dshm\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.284674 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.284577 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-home\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.284674 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.284622 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-model-cache\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.284796 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.284779 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b9f088-a105-4af4-82ad-3efdf214a5e2-tls-certs\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.284851 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.284809 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh5fl\" (UniqueName: \"kubernetes.io/projected/88b9f088-a105-4af4-82ad-3efdf214a5e2-kube-api-access-bh5fl\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.292056 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.292033 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2d9861\"/\"kube-root-ca.crt\"" Oct 30 19:41:17.292190 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.292117 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2d9861\"/\"openshift-service-ca.crt\"" Oct 30 19:41:17.297346 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.297324 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2d9861\"/\"default-dockercfg-4lbfj\"" Oct 30 19:41:17.297478 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.297460 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2d9861\"/\"distinct-w0-2d9861-kserve-self-signed-certs\"" Oct 30 19:41:17.301838 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.301816 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k"] Oct 30 19:41:17.394016 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.386695 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-tmp-dir\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.394337 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.394301 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-dshm\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.394642 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.394622 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-home\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.409589 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.409544 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-model-cache\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.409774 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.409726 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b9f088-a105-4af4-82ad-3efdf214a5e2-tls-certs\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.409774 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.409760 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh5fl\" (UniqueName: \"kubernetes.io/projected/88b9f088-a105-4af4-82ad-3efdf214a5e2-kube-api-access-bh5fl\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.409883 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.409827 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-model-cache\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.409883 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.409443 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-home\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.409883 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.395159 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-tmp-dir\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.410780 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.410754 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-dshm\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.413684 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.413664 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b9f088-a105-4af4-82ad-3efdf214a5e2-tls-certs\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.448126 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.448096 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:17.471395 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.471364 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh5fl\" (UniqueName: \"kubernetes.io/projected/88b9f088-a105-4af4-82ad-3efdf214a5e2-kube-api-access-bh5fl\") pod \"distinct-w0-2d9861-kserve-5976675cd7-6qd7k\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.547001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.546895 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:17.688003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.684432 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:17.738716 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.738676 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" event={"ID":"abe0d869-8b60-4ef3-acf4-8a0c2375b35c","Type":"ContainerStarted","Data":"d7d53f608a0c73cd0cc76fef46666a61c24cc30e2a9593db5e6cb6e5da7e8008"} Oct 30 19:41:17.739076 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.739042 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" event={"ID":"abe0d869-8b60-4ef3-acf4-8a0c2375b35c","Type":"ContainerStarted","Data":"06e6623acfed759fd5eba22bf5e31d175d568452797fd5f4f66ac414db8bfac4"} Oct 30 19:41:17.862009 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.861947 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" podStartSLOduration=1.861935289 podStartE2EDuration="1.861935289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:17.858911451 +0000 UTC m=+1524.793698975" watchObservedRunningTime="2026-10-30 19:41:17.861935289 +0000 UTC m=+1524.796722812" Oct 30 19:41:17.978479 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:17.978447 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k"] Oct 30 19:41:17.979873 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:41:17.979848 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88b9f088_a105_4af4_82ad_3efdf214a5e2.slice/crio-d7da3d506c29c2cc9fe67b5deb1571ebc54187866300e359e59a4f940a375232 WatchSource:0}: Error finding container d7da3d506c29c2cc9fe67b5deb1571ebc54187866300e359e59a4f940a375232: Status 404 returned error can't find the container with id d7da3d506c29c2cc9fe67b5deb1571ebc54187866300e359e59a4f940a375232 Oct 30 19:41:18.446238 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:18.446205 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:18.448114 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:18.448085 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:41:18.448230 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:18.448144 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" podUID="abe0d869-8b60-4ef3-acf4-8a0c2375b35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 19:41:18.747533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:18.747452 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" event={"ID":"88b9f088-a105-4af4-82ad-3efdf214a5e2","Type":"ContainerStarted","Data":"9f14f36cd46a72086207af110f0522bfdb9fca4fe26e3bf65ed978e3dfc3f5fe"} Oct 30 19:41:18.747533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:18.747505 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" event={"ID":"88b9f088-a105-4af4-82ad-3efdf214a5e2","Type":"ContainerStarted","Data":"d7da3d506c29c2cc9fe67b5deb1571ebc54187866300e359e59a4f940a375232"} Oct 30 19:41:18.876943 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:18.876886 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" podStartSLOduration=1.8768692420000002 podStartE2EDuration="1.876869242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:18.874714491 +0000 UTC m=+1525.809502022" watchObservedRunningTime="2026-10-30 19:41:18.876869242 +0000 UTC m=+1525.811656767" Oct 30 19:41:19.446306 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:19.446270 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:41:19.446793 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:19.446339 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" podUID="abe0d869-8b60-4ef3-acf4-8a0c2375b35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 19:41:19.547439 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:19.547407 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:20.446279 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:20.446246 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:41:20.446453 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:20.446312 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" podUID="abe0d869-8b60-4ef3-acf4-8a0c2375b35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 19:41:21.446032 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:21.445974 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:41:21.446206 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:21.446060 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" podUID="abe0d869-8b60-4ef3-acf4-8a0c2375b35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 19:41:22.269183 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:22.269148 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:22.446176 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:22.446141 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:41:22.446347 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:22.446194 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" podUID="abe0d869-8b60-4ef3-acf4-8a0c2375b35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 19:41:23.368449 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:41:23.368408 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:41:23.450230 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:23.450195 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:23.450567 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:23.450544 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:23.451164 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:23.451140 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:41:27.206757 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:27.206724 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:27.548039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:27.547940 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:27.560927 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:27.560899 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:27.812651 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:27.812630 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:41:29.444423 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:29.444387 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:29.679629 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:29.679597 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:32.152573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:32.152536 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:36.338697 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:36.338660 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:37.044739 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:37.044686 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:41.711571 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:41.711528 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:46.209994 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.209947 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9"] Oct 30 19:41:46.213730 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.213714 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.267753 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.267723 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.267893 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.267869 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.267938 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.267906 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.267972 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.267955 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.268093 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.268081 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdkrt\" (UniqueName: \"kubernetes.io/projected/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-kube-api-access-hdkrt\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.268158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.268143 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.268210 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.268187 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.268318 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.268298 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.268385 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.268363 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.315809 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.315772 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9"] Oct 30 19:41:46.369921 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.369886 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.370089 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.369965 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.370089 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.370009 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.370089 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.370057 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.370218 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.370087 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdkrt\" (UniqueName: \"kubernetes.io/projected/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-kube-api-access-hdkrt\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.370218 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.370169 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.370322 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.370224 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.370322 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.370282 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.370416 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.370360 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.370472 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.370453 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.370568 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.370544 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.370717 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.370676 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.370779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.370718 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.371041 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.371020 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.372280 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.372252 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.372490 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.372473 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.408870 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.408838 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:46.427457 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.427413 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.427664 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.427640 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdkrt\" (UniqueName: \"kubernetes.io/projected/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-kube-api-access-hdkrt\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.527749 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.527665 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:46.957182 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.957131 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9"] Oct 30 19:41:46.958164 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:41:46.958138 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34bbe99d_caf6_46cb_a4b4_622ee14e8d75.slice/crio-48f67ed9466bd6006cd619cefb7924d7b1d3d412a3d39652c66202e4451a217b WatchSource:0}: Error finding container 48f67ed9466bd6006cd619cefb7924d7b1d3d412a3d39652c66202e4451a217b: Status 404 returned error can't find the container with id 48f67ed9466bd6006cd619cefb7924d7b1d3d412a3d39652c66202e4451a217b Oct 30 19:41:46.960383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.960357 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:41:46.960463 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.960424 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:41:46.960463 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:46.960453 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:41:47.939327 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:47.939285 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" event={"ID":"34bbe99d-caf6-46cb-a4b4-622ee14e8d75","Type":"ContainerStarted","Data":"6d91a3053fcdd29137f4458870a68b866b73eaa52c0bc7944fe970c73952ec8d"} Oct 30 19:41:47.939835 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:47.939355 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" event={"ID":"34bbe99d-caf6-46cb-a4b4-622ee14e8d75","Type":"ContainerStarted","Data":"48f67ed9466bd6006cd619cefb7924d7b1d3d412a3d39652c66202e4451a217b"} Oct 30 19:41:48.528077 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:48.528041 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:48.529398 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:48.529372 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:41:48.529513 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:48.529421 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" podUID="34bbe99d-caf6-46cb-a4b4-622ee14e8d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 19:41:49.528937 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:49.528898 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:41:49.529378 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:49.529013 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" podUID="34bbe99d-caf6-46cb-a4b4-622ee14e8d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 19:41:50.410489 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:50.410455 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:50.528722 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:50.528686 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:41:50.528892 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:50.528737 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" podUID="34bbe99d-caf6-46cb-a4b4-622ee14e8d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 19:41:51.099645 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:51.099574 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:51.528513 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:51.528421 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:41:51.528513 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:51.528488 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" podUID="34bbe99d-caf6-46cb-a4b4-622ee14e8d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 19:41:52.528053 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:52.528019 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:41:52.528465 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:52.528087 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" podUID="34bbe99d-caf6-46cb-a4b4-622ee14e8d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 19:41:53.366553 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:41:53.366518 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:41:53.528806 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:53.528772 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:41:53.529198 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:53.528834 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" podUID="34bbe99d-caf6-46cb-a4b4-622ee14e8d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 19:41:54.532380 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:54.532351 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:54.532828 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:54.532814 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:54.533370 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:54.533351 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:41:54.701994 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:54.701937 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" podStartSLOduration=8.701924669 podStartE2EDuration="8.701924669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:48.173178136 +0000 UTC m=+1555.107965659" watchObservedRunningTime="2026-10-30 19:41:54.701924669 +0000 UTC m=+1561.636712203" Oct 30 19:41:55.880637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:41:55.880531 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:00.534355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:00.534305 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:01.509868 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.509835 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn"] Oct 30 19:42:01.515129 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.515109 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.571869 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.571836 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.572237 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.571879 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.572237 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.571903 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.572237 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.572068 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.572237 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.572146 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.572237 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.572194 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.572397 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.572262 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a62b512-6b1a-4974-8512-eb702c044ecd-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.572397 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.572283 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqbps\" (UniqueName: \"kubernetes.io/projected/2a62b512-6b1a-4974-8512-eb702c044ecd-kube-api-access-tqbps\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.572397 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.572353 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.605022 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.604005 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn"] Oct 30 19:42:01.673338 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.673307 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.673514 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.673364 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.673514 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.673418 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a62b512-6b1a-4974-8512-eb702c044ecd-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.673514 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.673446 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqbps\" (UniqueName: \"kubernetes.io/projected/2a62b512-6b1a-4974-8512-eb702c044ecd-kube-api-access-tqbps\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.673514 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.673496 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.673717 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.673582 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.673717 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.673623 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.673717 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.673659 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.673864 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.673722 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.673864 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.673789 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.673970 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.673870 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.673970 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.673939 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.674231 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.674210 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.674308 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.674262 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a62b512-6b1a-4974-8512-eb702c044ecd-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.675998 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.675966 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.676270 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.676249 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.731461 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.731429 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.731692 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.731542 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqbps\" (UniqueName: \"kubernetes.io/projected/2a62b512-6b1a-4974-8512-eb702c044ecd-kube-api-access-tqbps\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:01.829807 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:01.829769 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:02.288622 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:02.285781 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn"] Oct 30 19:42:02.295533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:02.293893 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:02.295533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:02.294001 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:02.295533 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:02.294047 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:03.109770 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:03.109727 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" event={"ID":"2a62b512-6b1a-4974-8512-eb702c044ecd","Type":"ContainerStarted","Data":"5d5c9928154f9ec145404b27612622513a1a6f974819ec0073635822c452e4c1"} Oct 30 19:42:03.110400 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:03.110372 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" event={"ID":"2a62b512-6b1a-4974-8512-eb702c044ecd","Type":"ContainerStarted","Data":"2e7111d135e462529dd431a8d9f81660aa83dccbb746bec6d3620a62bbe68409"} Oct 30 19:42:03.347571 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:03.347500 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" podStartSLOduration=2.347476301 podStartE2EDuration="2.347476301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:03.296249485 +0000 UTC m=+1570.231037010" watchObservedRunningTime="2026-10-30 19:42:03.347476301 +0000 UTC m=+1570.282263826" Oct 30 19:42:03.881002 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:03.877820 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:03.892019 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:03.891144 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:42:03.892019 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:03.891237 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" podUID="2a62b512-6b1a-4974-8512-eb702c044ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:42:03.901151 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:03.901113 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" podUID="6c890add-9063-4067-bb5e-65324cf99b3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:03.914181 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:03.914135 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" podUID="c2ffbe2b-c076-4ae7-994b-d8ab2db02185" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 30 19:42:03.914640 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:03.914613 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" podUID="ee9f5fbf-a71a-491e-abd3-8b59a15066ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": context deadline exceeded" Oct 30 19:42:03.931744 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:03.931704 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:03.952545 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:03.952492 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" podUID="7f1295b7-8682-4eed-b904-83b8b6537e8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:04.033003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:04.024104 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" podUID="c2ffbe2b-c076-4ae7-994b-d8ab2db02185" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:04.172074 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:04.164789 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:42:04.205135 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:04.205064 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" podUID="6c890add-9063-4067-bb5e-65324cf99b3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Oct 30 19:42:04.444374 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:04.444232 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" podUID="0d290229-7e04-47a2-8c0a-c18988cd07f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:04.486137 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:04.486066 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:04.832003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:04.830502 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:42:04.832003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:04.830588 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" podUID="2a62b512-6b1a-4974-8512-eb702c044ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:42:05.098088 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:05.080342 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 19:42:05.098088 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:05.084100 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:42:05.167173 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:05.167125 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:42:05.168311 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:05.167822 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:42:05.469012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:05.468869 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" podUID="0d290229-7e04-47a2-8c0a-c18988cd07f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:05.651465 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:05.651412 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:05.668164 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:05.664301 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" podUID="eafb6a50-b7db-47d6-b742-86b92bfef67d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 19:42:05.668996 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:05.668656 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:42:05.669269 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:05.669254 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:42:05.669430 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:05.669414 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:42:05.669587 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:05.669577 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:42:05.678095 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:05.678055 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:42:05.680744 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:05.680636 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:42:05.746455 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:05.746224 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:05.830727 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:05.830695 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:42:05.830901 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:05.830754 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" podUID="2a62b512-6b1a-4974-8512-eb702c044ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:42:06.830783 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:06.830751 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:42:06.831205 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:06.830813 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" podUID="2a62b512-6b1a-4974-8512-eb702c044ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:42:07.240040 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:07.239920 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:07.831070 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:07.831039 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:42:07.831531 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:07.831492 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" podUID="2a62b512-6b1a-4974-8512-eb702c044ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:42:08.830867 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:08.830834 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:42:08.831053 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:08.830886 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" podUID="2a62b512-6b1a-4974-8512-eb702c044ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:42:09.830364 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:09.830324 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:42:09.830775 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:09.830378 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" podUID="2a62b512-6b1a-4974-8512-eb702c044ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:42:10.843353 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:10.843318 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:10.844219 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:10.844196 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:10.845259 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:10.845237 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:42:11.845189 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:11.845145 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:15.485378 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:15.485337 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:17.737157 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:17.737111 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:21.368570 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.368537 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:21.467379 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.467347 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw"] Oct 30 19:42:21.471487 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.471469 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.567687 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.567658 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw"] Oct 30 19:42:21.601327 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.601293 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.601463 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.601349 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.601463 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.601433 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.601575 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.601488 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.601628 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.601605 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.601682 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.601671 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.601743 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.601724 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.601850 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.601828 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwl8v\" (UniqueName: \"kubernetes.io/projected/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-kube-api-access-bwl8v\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.601919 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.601868 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.702851 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.702780 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.702851 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.702816 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.702851 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.702838 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.703098 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.702951 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.703098 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.703036 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.703098 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.703090 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.703233 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.703125 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.703233 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.703202 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwl8v\" (UniqueName: \"kubernetes.io/projected/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-kube-api-access-bwl8v\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.703327 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.703233 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.703327 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.703265 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.703327 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.703297 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.703510 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.703486 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.703621 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.703600 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.703740 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.703723 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.705298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.705270 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.705444 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.705427 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.760043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.760015 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.760228 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.760204 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwl8v\" (UniqueName: \"kubernetes.io/projected/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-kube-api-access-bwl8v\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:21.790108 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:21.790078 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:22.237941 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:22.237914 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw"] Oct 30 19:42:22.239884 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:42:22.239853 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a48e18_31cf_4e6b_acdb_4f4d9d93d9f3.slice/crio-83b0db5054b353aace5c7e7ffe8e374eb506428de1b3b2750172b5113156b9e3 WatchSource:0}: Error finding container 83b0db5054b353aace5c7e7ffe8e374eb506428de1b3b2750172b5113156b9e3: Status 404 returned error can't find the container with id 83b0db5054b353aace5c7e7ffe8e374eb506428de1b3b2750172b5113156b9e3 Oct 30 19:42:22.242377 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:22.242343 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:22.242480 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:22.242422 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:22.242480 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:22.242467 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:22.724478 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:22.724441 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" event={"ID":"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3","Type":"ContainerStarted","Data":"eb2353eeb6fe16757e609eefeabbdc4fb1de4b5d5be097cdeb83e32cc962d013"} Oct 30 19:42:22.724911 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:22.724506 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" event={"ID":"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3","Type":"ContainerStarted","Data":"83b0db5054b353aace5c7e7ffe8e374eb506428de1b3b2750172b5113156b9e3"} Oct 30 19:42:22.800913 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:22.800880 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:22.844186 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:22.844125 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" podStartSLOduration=1.844108836 podStartE2EDuration="1.844108836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:22.842973655 +0000 UTC m=+1589.777761180" watchObservedRunningTime="2026-10-30 19:42:22.844108836 +0000 UTC m=+1589.778896360" Oct 30 19:42:23.366237 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:42:23.366207 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:42:23.810173 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:23.810076 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:42:23.810173 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:23.810134 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" podUID="07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:42:24.809004 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:24.804298 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:42:24.809004 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:24.804356 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" podUID="07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:42:25.809381 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:25.809347 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:42:25.809853 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:25.809494 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" podUID="07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:42:25.912505 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:25.912474 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5"] Oct 30 19:42:25.916166 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:25.916146 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:25.955730 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:25.955704 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367229e1-f694-4449-9e91-eadaf2502938-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:25.955850 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:25.955734 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fqnt\" (UniqueName: \"kubernetes.io/projected/367229e1-f694-4449-9e91-eadaf2502938-kube-api-access-5fqnt\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:25.955850 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:25.955753 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:25.955850 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:25.955823 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367229e1-f694-4449-9e91-eadaf2502938-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:25.956063 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:25.955878 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:25.956063 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:25.955903 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:25.956063 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:25.956012 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367229e1-f694-4449-9e91-eadaf2502938-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:25.956185 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:25.956113 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:25.956185 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:25.956134 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.015621 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.015585 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5"] Oct 30 19:42:26.057510 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.057470 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.057668 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.057536 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.057668 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.057605 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367229e1-f694-4449-9e91-eadaf2502938-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.057668 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.057631 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqnt\" (UniqueName: \"kubernetes.io/projected/367229e1-f694-4449-9e91-eadaf2502938-kube-api-access-5fqnt\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.057852 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.057666 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.057852 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.057725 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367229e1-f694-4449-9e91-eadaf2502938-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.057966 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.057871 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.058149 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.058121 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.058222 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.058208 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.058285 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.058237 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.058337 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.058317 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367229e1-f694-4449-9e91-eadaf2502938-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.058645 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.058581 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.058862 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.058836 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.059078 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.059060 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367229e1-f694-4449-9e91-eadaf2502938-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.060169 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.060097 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.060372 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.060347 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367229e1-f694-4449-9e91-eadaf2502938-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.121476 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.121445 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fqnt\" (UniqueName: \"kubernetes.io/projected/367229e1-f694-4449-9e91-eadaf2502938-kube-api-access-5fqnt\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.123842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.123814 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367229e1-f694-4449-9e91-eadaf2502938-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.229632 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.229602 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:26.242428 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.242399 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:26.715036 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.715007 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5"] Oct 30 19:42:26.717187 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.717158 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:26.717281 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.717235 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:26.717281 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.717264 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:26.755580 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.755558 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" event={"ID":"367229e1-f694-4449-9e91-eadaf2502938","Type":"ContainerStarted","Data":"5fa0a2cf2a548930f0877d7809412a2dbc97e0994bed16e1b83598808cf7dee0"} Oct 30 19:42:26.809001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.807202 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:42:26.809001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:26.807254 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" podUID="07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:42:27.763667 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:27.763623 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" event={"ID":"367229e1-f694-4449-9e91-eadaf2502938","Type":"ContainerStarted","Data":"c034c45611b330f8e24a09b20f826204843342f85b43de54ff79a3f41715b897"} Oct 30 19:42:27.814003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:27.805253 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:42:27.814003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:27.805314 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" podUID="07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:42:27.972055 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:27.971999 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" podStartSLOduration=2.971973438 podStartE2EDuration="2.971973438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:27.96978887 +0000 UTC m=+1594.904576394" watchObservedRunningTime="2026-10-30 19:42:27.971973438 +0000 UTC m=+1594.906760961" Oct 30 19:42:28.230161 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:28.230124 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:28.231797 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:28.231774 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:42:28.231925 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:28.231816 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" podUID="367229e1-f694-4449-9e91-eadaf2502938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:42:28.809005 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:28.807172 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:42:28.809005 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:28.807239 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" podUID="07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:42:29.230942 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:29.230896 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:42:29.231160 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:29.230948 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" podUID="367229e1-f694-4449-9e91-eadaf2502938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:42:29.812645 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:29.812616 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:29.813093 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:29.813075 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:29.813703 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:29.813682 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:42:30.230287 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:30.230199 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:42:30.230287 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:30.230262 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" podUID="367229e1-f694-4449-9e91-eadaf2502938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:42:30.943693 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:30.943653 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:31.230821 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:31.230727 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:42:31.230821 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:31.230773 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" podUID="367229e1-f694-4449-9e91-eadaf2502938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:42:32.230172 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:32.230138 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:42:32.230548 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:32.230188 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" podUID="367229e1-f694-4449-9e91-eadaf2502938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:42:33.230662 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:33.230622 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:42:33.231205 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:33.230672 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" podUID="367229e1-f694-4449-9e91-eadaf2502938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:42:34.234005 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:34.233963 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:34.234458 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:34.234245 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:34.235010 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:34.234969 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:42:35.867438 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:35.867402 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:37.662004 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:37.658053 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:42:37.686000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:37.682385 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 19:42:37.686000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:37.683965 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" podUID="0d290229-7e04-47a2-8c0a-c18988cd07f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 19:42:37.864004 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:37.863123 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" podUID="6c890add-9063-4067-bb5e-65324cf99b3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:37.965065 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:37.964804 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" podUID="7f1295b7-8682-4eed-b904-83b8b6537e8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:38.022004 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.021421 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" podUID="c2ffbe2b-c076-4ae7-994b-d8ab2db02185" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:38.524004 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.523362 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 19:42:38.557169 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.557125 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded" Oct 30 19:42:38.573793 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.572339 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" podUID="ee9f5fbf-a71a-491e-abd3-8b59a15066ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:38.573793 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.572728 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 19:42:38.632005 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.628349 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" podUID="eafb6a50-b7db-47d6-b742-86b92bfef67d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:38.683295 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.662071 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:38.683295 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.662451 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:42:38.703001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.698464 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" podUID="0d290229-7e04-47a2-8c0a-c18988cd07f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:38.703001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.698835 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 19:42:38.703001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.699173 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:42:38.703001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.699228 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:42:38.835065 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.819535 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:38.867304 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.866101 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" podUID="6c890add-9063-4067-bb5e-65324cf99b3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:38.867304 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.866505 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:42:38.979003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.978177 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" podUID="7f1295b7-8682-4eed-b904-83b8b6537e8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:38.979003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:38.978558 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:42:39.594636 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:39.594584 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" podUID="ee9f5fbf-a71a-491e-abd3-8b59a15066ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 19:42:39.595217 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:39.595186 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 19:42:39.656053 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:39.653387 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" podUID="eafb6a50-b7db-47d6-b742-86b92bfef67d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:39.690524 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:39.690477 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:39.702195 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:39.702153 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 19:42:39.728002 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:39.702659 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" podUID="0d290229-7e04-47a2-8c0a-c18988cd07f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:39.805170 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:39.805121 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:42:39.805481 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:39.805431 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" podUID="75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:39.898078 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:39.890408 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" podUID="6c890add-9063-4067-bb5e-65324cf99b3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:40.001315 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:40.000719 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" podUID="7f1295b7-8682-4eed-b904-83b8b6537e8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 19:42:40.008180 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:40.004432 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:42:40.181202 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:40.181098 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" podUID="c2ffbe2b-c076-4ae7-994b-d8ab2db02185" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:40.217725 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:40.217677 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" podUID="0d290229-7e04-47a2-8c0a-c18988cd07f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:40.236004 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:40.045231 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:42:40.236004 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:40.222156 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:42:40.236004 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:40.222194 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:42:40.381024 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:40.370956 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:42:40.561063 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:40.548406 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:42:40.611491 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:40.610145 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 19:42:40.691739 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:40.691710 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:42:40.699663 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:40.692929 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:42:40.725293 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:40.709207 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:42:40.725293 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:40.712054 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:40.746630 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:40.746605 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:42:40.755100 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:40.755082 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:42:42.717093 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:42.717056 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:47.419537 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:47.419471 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:50.734513 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:50.734476 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:51.721985 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.721943 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m"] Oct 30 19:42:51.726412 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.726392 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.830507 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.830484 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m"] Oct 30 19:42:51.832222 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.832201 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.832304 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.832256 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.832371 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.832299 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svf2w\" (UniqueName: \"kubernetes.io/projected/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-kube-api-access-svf2w\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.832432 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.832403 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.832487 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.832456 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.832536 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.832485 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.832587 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.832543 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.832677 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.832662 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.832743 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.832689 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.934376 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.934346 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.934529 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.934387 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.934529 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.934498 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.934529 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.934538 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.934750 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.934615 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.934750 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.934635 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.934750 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.934717 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.934924 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.934801 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.934924 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.934824 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.934924 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.934828 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svf2w\" (UniqueName: \"kubernetes.io/projected/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-kube-api-access-svf2w\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.935262 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.935019 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.935262 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.935173 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.935262 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.935192 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.935528 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.935507 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.936513 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.936493 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.936778 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.936761 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.998329 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.998255 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:51.998505 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:51.998379 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svf2w\" (UniqueName: \"kubernetes.io/projected/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-kube-api-access-svf2w\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:52.043164 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:52.040273 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:52.043164 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:52.042824 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:52.497659 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:52.496950 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m"] Oct 30 19:42:52.499929 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:42:52.499899 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9ef6f1_ed94_474c_9da2_d2cc68f3ec8a.slice/crio-1d48ab511f92bad0fbde10d88ee38658191c0973e1b7c4d5ef406f609a02fe56 WatchSource:0}: Error finding container 1d48ab511f92bad0fbde10d88ee38658191c0973e1b7c4d5ef406f609a02fe56: Status 404 returned error can't find the container with id 1d48ab511f92bad0fbde10d88ee38658191c0973e1b7c4d5ef406f609a02fe56 Oct 30 19:42:52.502466 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:52.502433 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:52.502553 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:52.502497 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:52.502553 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:52.502526 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:53.366537 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:42:53.366505 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:42:53.374461 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:53.374434 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" event={"ID":"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a","Type":"ContainerStarted","Data":"c12f10c5298f6bda829aae5abe4395c7c9d153f47c83ae85f679ed7317396c03"} Oct 30 19:42:53.374590 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:53.374488 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" event={"ID":"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a","Type":"ContainerStarted","Data":"1d48ab511f92bad0fbde10d88ee38658191c0973e1b7c4d5ef406f609a02fe56"} Oct 30 19:42:53.566308 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:53.566259 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" podStartSLOduration=2.566246297 podStartE2EDuration="2.566246297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:53.541701699 +0000 UTC m=+1620.476489224" watchObservedRunningTime="2026-10-30 19:42:53.566246297 +0000 UTC m=+1620.501033821" Oct 30 19:42:54.046019 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:54.045988 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:42:54.047723 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:54.047699 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:42:54.047844 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:54.047746 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" podUID="5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:42:55.046460 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:55.046425 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:42:55.046851 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:55.046475 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" podUID="5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:42:56.047034 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:56.046957 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:42:56.047415 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:56.047060 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" podUID="5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:42:56.925900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:56.925858 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:57.046220 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:57.046186 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:42:57.046384 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:57.046240 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" podUID="5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:42:58.046045 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:58.046012 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:42:58.046417 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:58.046063 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" podUID="5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:42:59.046080 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:59.046044 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:42:59.046446 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:42:59.046104 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" podUID="5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:43:00.050735 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:00.050705 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:43:00.051231 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:00.051214 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:43:00.051906 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:00.051887 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:43:01.575858 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.575819 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:01.806021 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.805975 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm"] Oct 30 19:43:01.811469 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.811447 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.869431 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.869379 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm"] Oct 30 19:43:01.890338 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.890313 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.890466 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.890357 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.890466 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.890380 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.890581 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.890497 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.890581 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.890543 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.890683 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.890597 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e60493-90ad-4718-a5a5-9acb15f7af19-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.890683 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.890618 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4nlm\" (UniqueName: \"kubernetes.io/projected/92e60493-90ad-4718-a5a5-9acb15f7af19-kube-api-access-h4nlm\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.890787 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.890679 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.890787 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.890760 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.992035 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.992009 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.992178 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.992078 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.992178 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.992109 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.992178 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.992127 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.992295 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.992271 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.992375 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.992360 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.992420 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.992405 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e60493-90ad-4718-a5a5-9acb15f7af19-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.992471 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.992421 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.992520 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.992471 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.992520 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.992496 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.992633 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.992524 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4nlm\" (UniqueName: \"kubernetes.io/projected/92e60493-90ad-4718-a5a5-9acb15f7af19-kube-api-access-h4nlm\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.992633 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.992558 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.992735 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.992668 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.992921 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.992904 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e60493-90ad-4718-a5a5-9acb15f7af19-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.994559 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.994536 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:01.994865 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:01.994849 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:02.054625 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:02.054599 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:02.077060 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:02.077032 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4nlm\" (UniqueName: \"kubernetes.io/projected/92e60493-90ad-4718-a5a5-9acb15f7af19-kube-api-access-h4nlm\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:02.124831 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:02.124776 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:02.552948 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:02.552924 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm"] Oct 30 19:43:02.553828 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:43:02.553797 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e60493_90ad_4718_a5a5_9acb15f7af19.slice/crio-c0fc33e4c6dead84bce80ced69441389caecadf2c053aab3deb9af383b030da5 WatchSource:0}: Error finding container c0fc33e4c6dead84bce80ced69441389caecadf2c053aab3deb9af383b030da5: Status 404 returned error can't find the container with id c0fc33e4c6dead84bce80ced69441389caecadf2c053aab3deb9af383b030da5 Oct 30 19:43:02.556030 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:02.556001 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:43:02.556100 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:02.556061 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:43:02.556100 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:02.556087 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:43:03.469807 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:03.469769 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" event={"ID":"92e60493-90ad-4718-a5a5-9acb15f7af19","Type":"ContainerStarted","Data":"7f630d4df21927d285584e21cadf7fa0ea9fe4d58466f8972b8e37dd8a97eeeb"} Oct 30 19:43:03.470262 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:03.469832 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" event={"ID":"92e60493-90ad-4718-a5a5-9acb15f7af19","Type":"ContainerStarted","Data":"c0fc33e4c6dead84bce80ced69441389caecadf2c053aab3deb9af383b030da5"} Oct 30 19:43:03.688006 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:03.685346 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" podStartSLOduration=2.6853280699999997 podStartE2EDuration="2.68532807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:03.63489303 +0000 UTC m=+1630.569680552" watchObservedRunningTime="2026-10-30 19:43:03.68532807 +0000 UTC m=+1630.620115596" Oct 30 19:43:04.125555 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:04.125516 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:04.127237 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:04.127209 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:43:04.127395 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:04.127273 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" podUID="92e60493-90ad-4718-a5a5-9acb15f7af19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:43:05.125799 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:05.125765 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:43:05.126266 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:05.125833 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" podUID="92e60493-90ad-4718-a5a5-9acb15f7af19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:43:05.511916 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:05.511829 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:06.127148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.125525 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:43:06.127148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.125622 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" podUID="92e60493-90ad-4718-a5a5-9acb15f7af19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:43:06.444225 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.441043 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9"] Oct 30 19:43:06.446676 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.446649 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.481586 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.481068 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:06.553269 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.553231 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-model-cache\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.553269 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.553268 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-tls-certs\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.553536 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.553512 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-dshm\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.553649 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.553634 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57hqz\" (UniqueName: \"kubernetes.io/projected/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-kube-api-access-57hqz\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.553760 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.553746 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-tmp-dir\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.553881 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.553862 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-home\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.554545 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.554525 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9"] Oct 30 19:43:06.568647 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.568624 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ada8d7\"/\"distinct-2-w3-ada8d7-kserve-self-signed-certs\"" Oct 30 19:43:06.654861 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.654820 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-dshm\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.655077 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.654883 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57hqz\" (UniqueName: \"kubernetes.io/projected/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-kube-api-access-57hqz\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.655077 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.654925 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-tmp-dir\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.655077 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.655027 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-home\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.655257 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.655125 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-model-cache\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.655257 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.655147 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-tls-certs\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.655423 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.655400 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-tmp-dir\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.655539 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.655447 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-home\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.655539 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.655480 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-model-cache\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.657419 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.657394 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-dshm\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.657665 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.657649 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-tls-certs\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.724160 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.724087 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57hqz\" (UniqueName: \"kubernetes.io/projected/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-kube-api-access-57hqz\") pod \"distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:06.734453 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.734422 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:06.761456 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:06.761429 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:07.125637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:07.125591 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:43:07.125814 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:07.125655 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" podUID="92e60493-90ad-4718-a5a5-9acb15f7af19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:43:07.213486 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:43:07.213456 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bd1c66b_4d08_4aca_ab6b_05b7267e7821.slice/crio-3567244eb522388b4bebf1bab633caeb81a9bc770e84b37cbff8efa20bf07d0c WatchSource:0}: Error finding container 3567244eb522388b4bebf1bab633caeb81a9bc770e84b37cbff8efa20bf07d0c: Status 404 returned error can't find the container with id 3567244eb522388b4bebf1bab633caeb81a9bc770e84b37cbff8efa20bf07d0c Oct 30 19:43:07.222415 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:07.222384 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9"] Oct 30 19:43:07.536692 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:07.536600 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" event={"ID":"1bd1c66b-4d08-4aca-ab6b-05b7267e7821","Type":"ContainerStarted","Data":"95b8e7d5e3dc77cd6c1c00ac9d5d4cdb28d0f0195e44ce0c35163d326614cfa0"} Oct 30 19:43:07.536692 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:07.536640 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" event={"ID":"1bd1c66b-4d08-4aca-ab6b-05b7267e7821","Type":"ContainerStarted","Data":"3567244eb522388b4bebf1bab633caeb81a9bc770e84b37cbff8efa20bf07d0c"} Oct 30 19:43:07.673013 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:07.672930 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" podStartSLOduration=1.672913404 podStartE2EDuration="1.672913404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:07.671611877 +0000 UTC m=+1634.606399422" watchObservedRunningTime="2026-10-30 19:43:07.672913404 +0000 UTC m=+1634.607700927" Oct 30 19:43:07.761579 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:07.761551 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:08.125191 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:08.125160 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:43:08.125376 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:08.125207 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" podUID="92e60493-90ad-4718-a5a5-9acb15f7af19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:43:09.127695 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:09.127400 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:43:09.127695 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:09.127464 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" podUID="92e60493-90ad-4718-a5a5-9acb15f7af19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:43:10.140267 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:10.140233 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:10.141047 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:10.141021 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:10.141974 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:10.141954 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:43:11.399187 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:11.399151 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:11.738147 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:11.738051 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:11.747860 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:11.747206 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" podUID="7f1295b7-8682-4eed-b904-83b8b6537e8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:11.769788 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:11.769117 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" podUID="0d290229-7e04-47a2-8c0a-c18988cd07f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:16.285815 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:16.285783 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:16.762479 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:16.762453 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:16.774285 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:16.774263 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:17.664663 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:17.664635 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:21.062957 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:21.062921 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:23.366254 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:43:23.366219 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:43:25.811436 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:25.811393 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:29.049633 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:29.049588 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:29.306402 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:29.306325 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:29.843956 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:29.843921 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:30.498566 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:30.498523 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:35.176734 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:35.176651 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:36.372855 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:36.372819 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:36.659753 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:36.659665 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:39.996618 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:39.996577 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:41.307464 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:41.307428 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:41.596881 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:41.596844 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:42.093803 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:42.090557 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" podUID="7f1295b7-8682-4eed-b904-83b8b6537e8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:42.225011 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:42.223442 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:42.586192 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:42.586153 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" podUID="ee9f5fbf-a71a-491e-abd3-8b59a15066ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:42.598003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:42.596576 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:42.659103 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:42.658115 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" podUID="1bd1c66b-4d08-4aca-ab6b-05b7267e7821" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:42.693880 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:42.693146 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:42.707363 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:42.706327 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 19:43:42.714165 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:42.713488 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:42.751433 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:42.750101 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" podUID="0d290229-7e04-47a2-8c0a-c18988cd07f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 19:43:42.751433 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:42.750814 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" podUID="7f1295b7-8682-4eed-b904-83b8b6537e8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:42.885052 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:42.884712 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:43.035046 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:43.034997 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" podUID="c2ffbe2b-c076-4ae7-994b-d8ab2db02185" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 19:43:43.210060 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:43.163243 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" podUID="6c890add-9063-4067-bb5e-65324cf99b3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:43.442022 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:43.404583 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" podUID="eafb6a50-b7db-47d6-b742-86b92bfef67d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:43.664082 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:43.664039 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" podUID="1bd1c66b-4d08-4aca-ab6b-05b7267e7821" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:43.730369 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:43.730324 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:43.730633 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:43.687832 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" podUID="ee9f5fbf-a71a-491e-abd3-8b59a15066ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 19:43:43.795031 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:43.788142 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 19:43:43.946957 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:43.941831 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" podUID="7f1295b7-8682-4eed-b904-83b8b6537e8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 19:43:43.946957 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:43.941938 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" podUID="c2ffbe2b-c076-4ae7-994b-d8ab2db02185" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:43.958222 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:43.958181 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:44.004346 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.004312 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:43:44.004599 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.004561 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" podUID="666f2ff7-bd9c-4525-bdf9-bda82db33b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:44.035135 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.035102 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7"] Oct 30 19:43:44.043172 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.042459 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" podUID="c2ffbe2b-c076-4ae7-994b-d8ab2db02185" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:44.201044 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.200878 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:44.201044 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.200961 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:43:44.201044 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.201028 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:43:44.201322 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.201084 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:43:44.201322 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.201143 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:43:44.201322 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.201198 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:43:44.201322 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.201251 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:43:44.214020 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.210851 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7"] Oct 30 19:43:44.234418 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.234392 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:43:44.236140 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.236118 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.238504 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.238485 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:43:44.261997 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.257475 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:43:44.261997 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.258435 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:44.299997 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.298804 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:43:44.346112 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.334968 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d74c1bd9-gw-openshift-default-dockercfg-kl4m6\"" Oct 30 19:43:44.366730 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.358001 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istiod-ca-cert\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.366730 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.358119 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-credential-socket\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.366730 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.358162 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-podinfo\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.366730 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.358375 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-envoy\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.366730 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.358422 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-token\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.366730 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.358814 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qphht\" (UniqueName: \"kubernetes.io/projected/44e31a3a-35aa-49a0-afd1-f722ecca6c08-kube-api-access-qphht\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.366730 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.359036 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-workload-socket\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.366730 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.359073 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-data\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.366730 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.359160 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-workload-certs\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.460107 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.460041 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:43:44.460407 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.460377 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qphht\" (UniqueName: \"kubernetes.io/projected/44e31a3a-35aa-49a0-afd1-f722ecca6c08-kube-api-access-qphht\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.460479 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.460465 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-workload-socket\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.460520 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.460489 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-data\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.460556 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.460532 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-workload-certs\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.460604 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.460578 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istiod-ca-cert\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.460646 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.460637 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-credential-socket\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.460684 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.460660 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-podinfo\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.460798 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.460780 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-envoy\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.460848 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.460826 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-token\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.464906 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.461436 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-workload-certs\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.475920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.462479 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:43:44.475920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.466562 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-workload-socket\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.475920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.466782 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-data\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.475920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.468229 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:43:44.475920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.470265 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istiod-ca-cert\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.475920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.469078 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-credential-socket\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.497908 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.497851 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-envoy\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.512238 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.499920 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-podinfo\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.537646 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.535161 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qphht\" (UniqueName: \"kubernetes.io/projected/44e31a3a-35aa-49a0-afd1-f722ecca6c08-kube-api-access-qphht\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.542736 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.542704 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-token\") pod \"e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.613792 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.613763 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:44.670367 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:44.670331 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:45.203637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:45.203602 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7"] Oct 30 19:43:45.214245 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:43:45.214207 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e31a3a_35aa_49a0_afd1_f722ecca6c08.slice/crio-048fd1f650cfb52a81bc15b14b74c89e9f05b14ed38594ddf530472f3a90ec4d WatchSource:0}: Error finding container 048fd1f650cfb52a81bc15b14b74c89e9f05b14ed38594ddf530472f3a90ec4d: Status 404 returned error can't find the container with id 048fd1f650cfb52a81bc15b14b74c89e9f05b14ed38594ddf530472f3a90ec4d Oct 30 19:43:45.216748 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:45.216715 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:43:45.216840 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:45.216780 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:43:45.216840 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:45.216823 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:43:45.273905 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:45.273879 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" event={"ID":"44e31a3a-35aa-49a0-afd1-f722ecca6c08","Type":"ContainerStarted","Data":"048fd1f650cfb52a81bc15b14b74c89e9f05b14ed38594ddf530472f3a90ec4d"} Oct 30 19:43:46.292007 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:46.290886 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" event={"ID":"44e31a3a-35aa-49a0-afd1-f722ecca6c08","Type":"ContainerStarted","Data":"4c3a9f2eaef934539ec36dc115f641b3fe8c8e098a2c7f3331db556f2678c221"} Oct 30 19:43:46.417806 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:46.417744 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" podStartSLOduration=3.41772647 podStartE2EDuration="3.41772647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:46.412627319 +0000 UTC m=+1673.347414845" watchObservedRunningTime="2026-10-30 19:43:46.41772647 +0000 UTC m=+1673.352513995" Oct 30 19:43:46.613946 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:46.613915 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:46.619569 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:46.619545 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:46.635452 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:46.635409 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:46.902916 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:46.902845 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:47.297069 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:47.296960 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:47.297920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:47.297902 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:43:47.398669 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:47.398628 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:48.036291 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:48.036252 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:49.114644 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:49.114597 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:50.753172 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:50.753137 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:51.341933 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:51.341886 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:51.726251 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:51.726160 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:51.986683 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:51.986569 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:53.369281 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:43:53.369245 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:43:54.381944 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:54.381909 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:54.707387 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:54.707308 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:54.912593 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:54.912556 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz"] Oct 30 19:43:54.916996 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:54.916962 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" Oct 30 19:43:54.970890 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:54.970807 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90239164-408c-440e-9b1d-d2cfc6745937-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz\" (UID: \"90239164-408c-440e-9b1d-d2cfc6745937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" Oct 30 19:43:54.971131 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:54.971108 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcgjf\" (UniqueName: \"kubernetes.io/projected/90239164-408c-440e-9b1d-d2cfc6745937-kube-api-access-gcgjf\") pod \"payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz\" (UID: \"90239164-408c-440e-9b1d-d2cfc6745937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" Oct 30 19:43:55.020876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:55.020846 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-94581ab2\"" Oct 30 19:43:55.021220 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:55.021198 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-94581ab2-dockercfg-prfsc\"" Oct 30 19:43:55.026817 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:55.026788 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz"] Oct 30 19:43:55.071874 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:55.071847 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcgjf\" (UniqueName: \"kubernetes.io/projected/90239164-408c-440e-9b1d-d2cfc6745937-kube-api-access-gcgjf\") pod \"payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz\" (UID: \"90239164-408c-440e-9b1d-d2cfc6745937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" Oct 30 19:43:55.072039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:55.071914 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90239164-408c-440e-9b1d-d2cfc6745937-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz\" (UID: \"90239164-408c-440e-9b1d-d2cfc6745937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" Oct 30 19:43:55.072684 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:55.072666 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90239164-408c-440e-9b1d-d2cfc6745937-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz\" (UID: \"90239164-408c-440e-9b1d-d2cfc6745937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" Oct 30 19:43:55.133775 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:55.133748 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcgjf\" (UniqueName: \"kubernetes.io/projected/90239164-408c-440e-9b1d-d2cfc6745937-kube-api-access-gcgjf\") pod \"payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz\" (UID: \"90239164-408c-440e-9b1d-d2cfc6745937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" Oct 30 19:43:55.231654 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:55.231594 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" Oct 30 19:43:55.615966 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:55.615929 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:55.618097 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:55.618069 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz"] Oct 30 19:43:55.619409 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:43:55.619382 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90239164_408c_440e_9b1d_d2cfc6745937.slice/crio-e1a94fe644165018ea6f04062d6414c9df813920c4db2c875a93d254b0073210 WatchSource:0}: Error finding container e1a94fe644165018ea6f04062d6414c9df813920c4db2c875a93d254b0073210: Status 404 returned error can't find the container with id e1a94fe644165018ea6f04062d6414c9df813920c4db2c875a93d254b0073210 Oct 30 19:43:56.060290 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:56.060191 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:56.357846 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:56.357812 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" event={"ID":"90239164-408c-440e-9b1d-d2cfc6745937","Type":"ContainerStarted","Data":"923d651c09d01e9dabc7c685086de5f9b20d625f0b34ff481acfd84a0f384b4f"} Oct 30 19:43:56.358037 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:56.357871 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" Oct 30 19:43:56.358037 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:56.357886 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" event={"ID":"90239164-408c-440e-9b1d-d2cfc6745937","Type":"ContainerStarted","Data":"e1a94fe644165018ea6f04062d6414c9df813920c4db2c875a93d254b0073210"} Oct 30 19:43:56.482136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:56.482091 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" podStartSLOduration=2.482077158 podStartE2EDuration="2.482077158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:56.480137219 +0000 UTC m=+1683.414924742" watchObservedRunningTime="2026-10-30 19:43:56.482077158 +0000 UTC m=+1683.416864703" Oct 30 19:43:57.012108 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:57.012075 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:57.391520 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:57.391491 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:57.702439 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:57.702358 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:58.457164 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:58.457130 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:58.798320 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:58.798249 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:59.638282 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:43:59.638247 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:02.390169 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:02.390125 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:02.664434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:02.664350 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:02.999457 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:02.999373 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:05.130520 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.130481 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv"] Oct 30 19:44:05.135070 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.135052 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.188830 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.188803 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 19:44:05.188962 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.188893 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 19:44:05.188962 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.188897 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 19:44:05.237747 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.237717 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2vd8g\"" Oct 30 19:44:05.243036 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.243009 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv"] Oct 30 19:44:05.243674 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.243651 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.243730 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.243714 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.243914 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.243894 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.244101 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.244077 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v758w\" (UniqueName: \"kubernetes.io/projected/bc78983a-aa83-48e3-b0c3-95148da6dfc1-kube-api-access-v758w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.244147 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.244121 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.244244 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.244222 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.244326 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.244308 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc78983a-aa83-48e3-b0c3-95148da6dfc1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.345973 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.345943 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc78983a-aa83-48e3-b0c3-95148da6dfc1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.346116 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.346021 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.346116 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.346067 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.346219 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.346163 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.346291 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.346275 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v758w\" (UniqueName: \"kubernetes.io/projected/bc78983a-aa83-48e3-b0c3-95148da6dfc1-kube-api-access-v758w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.346346 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.346315 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.346399 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.346364 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.347108 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.347082 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.348376 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.348346 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc78983a-aa83-48e3-b0c3-95148da6dfc1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.348481 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.348423 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.348557 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.348510 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.348607 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.348563 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.405199 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.405144 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v758w\" (UniqueName: \"kubernetes.io/projected/bc78983a-aa83-48e3-b0c3-95148da6dfc1-kube-api-access-v758w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.407068 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.407048 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pxwxv\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.447230 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.447212 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:05.878332 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.878304 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv"] Oct 30 19:44:05.879423 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:44:05.879382 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc78983a_aa83_48e3_b0c3_95148da6dfc1.slice/crio-d9e6db86fd0118ea67f0f58cb53a5e0b9e4e2fb1b965fe5d700c70ea58a19bed WatchSource:0}: Error finding container d9e6db86fd0118ea67f0f58cb53a5e0b9e4e2fb1b965fe5d700c70ea58a19bed: Status 404 returned error can't find the container with id d9e6db86fd0118ea67f0f58cb53a5e0b9e4e2fb1b965fe5d700c70ea58a19bed Oct 30 19:44:05.881373 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:05.881354 4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:44:06.434764 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:06.434734 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" event={"ID":"bc78983a-aa83-48e3-b0c3-95148da6dfc1","Type":"ContainerStarted","Data":"d9e6db86fd0118ea67f0f58cb53a5e0b9e4e2fb1b965fe5d700c70ea58a19bed"} Oct 30 19:44:06.735149 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:06.735078 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:07.367527 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.367498 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" Oct 30 19:44:07.547139 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.547100 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb"] Oct 30 19:44:07.556769 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.556742 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.615210 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.615178 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb"] Oct 30 19:44:07.670095 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.670024 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ada8d7-openshift-default-dockercfg-wtmwk\"" Oct 30 19:44:07.673660 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.673628 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a6ee20-cbc7-49e8-b581-19310616c0ee-istiod-ca-cert\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.673776 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.673675 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-workload-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.673926 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.673900 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-envoy\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.674076 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.674056 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-data\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.674173 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.674157 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-token\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.674238 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.674219 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-podinfo\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.674292 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.674280 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97cp8\" (UniqueName: \"kubernetes.io/projected/53a6ee20-cbc7-49e8-b581-19310616c0ee-kube-api-access-97cp8\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.674402 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.674383 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-workload-certs\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.674463 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.674446 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-credential-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.776070 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.776044 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-token\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.776188 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.776091 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-podinfo\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.776188 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.776126 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97cp8\" (UniqueName: \"kubernetes.io/projected/53a6ee20-cbc7-49e8-b581-19310616c0ee-kube-api-access-97cp8\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.776188 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.776150 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-workload-certs\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.776188 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.776182 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-credential-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.776492 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.776237 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a6ee20-cbc7-49e8-b581-19310616c0ee-istiod-ca-cert\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.776492 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.776262 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-workload-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.776492 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.776298 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-envoy\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.776492 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.776382 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-data\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.776687 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.776638 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-workload-certs\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.776750 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.776717 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-credential-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.776809 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.776756 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-data\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.776809 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.776788 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-workload-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.776993 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.776962 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a6ee20-cbc7-49e8-b581-19310616c0ee-istiod-ca-cert\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.778545 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.778524 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-envoy\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.778689 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.778671 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-podinfo\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.834352 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.834325 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-token\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.835722 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.835697 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97cp8\" (UniqueName: \"kubernetes.io/projected/53a6ee20-cbc7-49e8-b581-19310616c0ee-kube-api-access-97cp8\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.871959 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.871927 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:07.905563 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:07.905508 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:08.327624 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:08.327598 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb"] Oct 30 19:44:08.329360 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:44:08.329151 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a6ee20_cbc7_49e8_b581_19310616c0ee.slice/crio-2921c5c72cc04bb01b18427ba341b58a10b648384d44bdab53de0cf484ca76bf WatchSource:0}: Error finding container 2921c5c72cc04bb01b18427ba341b58a10b648384d44bdab53de0cf484ca76bf: Status 404 returned error can't find the container with id 2921c5c72cc04bb01b18427ba341b58a10b648384d44bdab53de0cf484ca76bf Oct 30 19:44:08.331910 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:08.331879 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:08.332032 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:08.331952 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:08.332101 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:08.332047 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:08.451419 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:08.451385 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" event={"ID":"53a6ee20-cbc7-49e8-b581-19310616c0ee","Type":"ContainerStarted","Data":"a377e6c5b990a9317e05e0fc45253e70ecc73cedf7d46e5237fb6ab1e96d65e3"} Oct 30 19:44:08.451595 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:08.451446 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" event={"ID":"53a6ee20-cbc7-49e8-b581-19310616c0ee","Type":"ContainerStarted","Data":"2921c5c72cc04bb01b18427ba341b58a10b648384d44bdab53de0cf484ca76bf"} Oct 30 19:44:08.578301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:08.578208 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" podStartSLOduration=1.578194492 podStartE2EDuration="1.578194492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:08.572417777 +0000 UTC m=+1695.507205300" watchObservedRunningTime="2026-10-30 19:44:08.578194492 +0000 UTC m=+1695.512982015" Oct 30 19:44:08.683178 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:08.683147 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:08.872290 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:08.872218 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:09.873949 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:09.873911 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:44:09.874422 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:09.873972 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" podUID="53a6ee20-cbc7-49e8-b581-19310616c0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 19:44:09.928707 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:09.928675 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:09.928795 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:09.928741 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:10.468700 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:10.468663 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" event={"ID":"bc78983a-aa83-48e3-b0c3-95148da6dfc1","Type":"ContainerStarted","Data":"fc51aea267f4fd3108796d990854fd6918dd14c543cde64583895b3cee7ea185"} Oct 30 19:44:10.468867 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:10.468746 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:10.872678 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:10.872643 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:44:10.872830 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:10.872706 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" podUID="53a6ee20-cbc7-49e8-b581-19310616c0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 19:44:11.113360 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:11.113326 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:11.479671 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:11.479635 4433 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-pxwxv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:44:11.479918 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:11.479898 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" podUID="bc78983a-aa83-48e3-b0c3-95148da6dfc1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:44:11.873074 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:11.873043 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:44:11.873270 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:11.873099 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" podUID="53a6ee20-cbc7-49e8-b581-19310616c0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 19:44:12.781504 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:12.781465 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:12.872295 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:12.872258 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:44:12.872463 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:12.872300 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" podUID="53a6ee20-cbc7-49e8-b581-19310616c0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 19:44:13.880001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:13.879660 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:44:13.880001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:13.879725 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" podUID="53a6ee20-cbc7-49e8-b581-19310616c0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 19:44:14.391903 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:14.391857 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" podUID="eafb6a50-b7db-47d6-b742-86b92bfef67d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:14.480839 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:14.480798 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 19:44:14.493424 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:14.493384 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 19:44:14.693975 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:14.693869 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:14.744409 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:14.744328 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" podStartSLOduration=5.698322929 podStartE2EDuration="9.744308631s" totalImagesPullingTime="4.045985702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:10.698605647 +0000 UTC m=+1697.633393172" watchObservedRunningTime="2026-10-30 19:44:14.744308631 +0000 UTC m=+1701.679096154" Oct 30 19:44:14.756281 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:14.756242 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" podUID="0d290229-7e04-47a2-8c0a-c18988cd07f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:14.897750 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:14.897716 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:44:14.910124 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:14.910081 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" podUID="53a6ee20-cbc7-49e8-b581-19310616c0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 19:44:15.115357 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.115301 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" podUID="6c890add-9063-4067-bb5e-65324cf99b3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:15.242052 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.242015 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" podUID="c2ffbe2b-c076-4ae7-994b-d8ab2db02185" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:15.274763 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.274042 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" podUID="ee9f5fbf-a71a-491e-abd3-8b59a15066ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:15.284042 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.278381 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:44:15.284356 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.284332 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" podUID="4ca58922-83fb-40bb-a289-1fcd9114ffd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:15.301615 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.301055 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" podUID="1bd1c66b-4d08-4aca-ab6b-05b7267e7821" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:15.432650 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.431066 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" podUID="eafb6a50-b7db-47d6-b742-86b92bfef67d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:15.448017 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.432958 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:44:15.481096 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.481056 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:15.490094 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.490057 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:15.509528 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.504210 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:44:15.612044 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.490290 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" podUID="7f1295b7-8682-4eed-b904-83b8b6537e8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:15.810112 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.809994 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 19:44:15.815001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.811852 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:44:15.938094 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.936687 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" podUID="0d290229-7e04-47a2-8c0a-c18988cd07f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 19:44:15.939503 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.939480 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:44:15.940680 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.940649 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:44:16.050031 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:15.959533 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" podUID="53a6ee20-cbc7-49e8-b581-19310616c0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 19:44:16.180219 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.154830 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:16.321001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.299659 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" podUID="6c890add-9063-4067-bb5e-65324cf99b3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 19:44:16.509353 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.322525 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" podUID="1bd1c66b-4d08-4aca-ab6b-05b7267e7821" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:16.520913 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.520883 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:44:16.521177 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.521159 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:44:16.546000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.523238 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:16.606348 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.395997 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" podUID="ee9f5fbf-a71a-491e-abd3-8b59a15066ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:16.609421 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.607660 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:44:16.609421 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.400462 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" podUID="c2ffbe2b-c076-4ae7-994b-d8ab2db02185" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:16.610115 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.433626 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" podUID="eafb6a50-b7db-47d6-b742-86b92bfef67d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:16.618029 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.610323 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:44:16.705001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.689070 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:16.731771 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.709128 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" podUID="7f1295b7-8682-4eed-b904-83b8b6537e8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:16.731771 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.712287 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:44:16.731771 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.712363 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:44:16.943942 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.943893 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:16.987393 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.971083 4433 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:44:16.987393 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.971151 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" podUID="53a6ee20-cbc7-49e8-b581-19310616c0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 19:44:16.987393 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:16.974260 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" podUID="0d290229-7e04-47a2-8c0a-c18988cd07f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:17.164754 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:17.164727 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 19:44:17.369686 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:17.369656 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 19:44:17.439698 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:17.438580 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 19:44:17.513702 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:17.513457 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 19:44:17.519683 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:17.519636 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" podUID="6c890add-9063-4067-bb5e-65324cf99b3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:17.520277 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:17.520149 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" podUID="1bd1c66b-4d08-4aca-ab6b-05b7267e7821" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:17.552201 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:17.552178 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 19:44:17.559378 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:17.559360 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 19:44:17.582010 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:17.581825 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 19:44:17.588049 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:17.585724 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 19:44:17.631040 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:17.630055 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" podUID="ee9f5fbf-a71a-491e-abd3-8b59a15066ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:17.796003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:17.795462 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 19:44:17.943049 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:17.942403 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk" Oct 30 19:44:18.303516 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:18.303440 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:18.350174 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:18.350043 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:18.350453 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:18.350431 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:18.351517 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:18.351493 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:44:18.899213 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:18.899183 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:21.474421 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:21.474380 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:23.017779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:23.017747 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:23.307675 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:23.307600 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:23.366317 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:44:23.366285 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:44:23.551435 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:23.551410 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:23.808141 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:23.808109 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:27.967059 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:27.966973 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:28.606102 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:28.606069 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:30.637487 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:30.637453 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:30.978171 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:30.978094 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:31.285957 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:31.285885 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:33.478503 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:33.478453 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:33.772785 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:33.772704 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:34.062311 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:34.062281 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:39.872334 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:39.872297 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:40.406412 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:40.406376 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:40.755178 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:40.755094 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:43.144231 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:43.144197 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:45.373225 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:45.373187 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:45.849250 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:45.849214 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:46.948149 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:46.948114 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:47.307380 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:47.307307 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:47.796515 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:47.796476 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x"] Oct 30 19:44:47.800837 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:47.800818 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:47.894569 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:47.894515 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x"] Oct 30 19:44:47.969481 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:47.969441 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:47.969943 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:47.969550 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:47.969943 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:47.969585 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:47.969943 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:47.969687 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww5n7\" (UniqueName: \"kubernetes.io/projected/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-kube-api-access-ww5n7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:47.969943 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:47.969810 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:47.969943 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:47.969853 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:47.969943 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:47.969897 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:47.969943 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:47.969929 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:47.970314 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:47.970124 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:47.983568 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:47.983537 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:48.071758 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.071723 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.071950 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.071851 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.071950 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.071880 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.072132 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.071995 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww5n7\" (UniqueName: \"kubernetes.io/projected/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-kube-api-access-ww5n7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.072132 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.072118 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.072240 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.072171 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.072240 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.072216 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.072340 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.072235 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.072340 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.072250 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.072340 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.072276 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.072510 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.072415 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.072702 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.072680 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.072856 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.072772 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.073253 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.073231 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.075102 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.075079 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.075418 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.075396 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.134255 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.134229 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.134484 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.134461 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww5n7\" (UniqueName: \"kubernetes.io/projected/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-kube-api-access-ww5n7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-pmt9x\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.418159 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.418080 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:48.438529 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.438475 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:48.871875 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.871841 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x"] Oct 30 19:44:48.910167 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:44:48.910134 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f198b8_0ce4_4ac4_9339_944e7f829bf1.slice/crio-0ca00d30d29ea9b44992f25603c3aa242c17fb4c4fe873612c95b371b858fc51 WatchSource:0}: Error finding container 0ca00d30d29ea9b44992f25603c3aa242c17fb4c4fe873612c95b371b858fc51: Status 404 returned error can't find the container with id 0ca00d30d29ea9b44992f25603c3aa242c17fb4c4fe873612c95b371b858fc51 Oct 30 19:44:48.912742 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.912709 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:48.912836 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.912791 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:48.912879 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:48.912838 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:49.576204 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:49.576169 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" event={"ID":"f7f198b8-0ce4-4ac4-9339-944e7f829bf1","Type":"ContainerStarted","Data":"a499b3bdacc88491e09d94c6eab2459bce23b99bfca2e320884e2605df336e56"} Oct 30 19:44:49.576616 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:49.576232 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" event={"ID":"f7f198b8-0ce4-4ac4-9339-944e7f829bf1","Type":"ContainerStarted","Data":"0ca00d30d29ea9b44992f25603c3aa242c17fb4c4fe873612c95b371b858fc51"} Oct 30 19:44:49.813999 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:49.806545 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" podStartSLOduration=2.80652699 podStartE2EDuration="2.80652699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:49.754092532 +0000 UTC m=+1736.688880056" watchObservedRunningTime="2026-10-30 19:44:49.80652699 +0000 UTC m=+1736.741314519" Oct 30 19:44:50.418931 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:50.418894 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:50.420928 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:50.420894 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pmt9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:44:50.421097 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:50.420959 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" podUID="f7f198b8-0ce4-4ac4-9339-944e7f829bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:44:50.554391 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:50.554363 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:51.419316 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:51.419282 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pmt9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:44:51.419690 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:51.419328 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" podUID="f7f198b8-0ce4-4ac4-9339-944e7f829bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:44:51.546270 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:51.546238 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:52.418389 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:52.418354 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pmt9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:44:52.418563 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:52.418406 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" podUID="f7f198b8-0ce4-4ac4-9339-944e7f829bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:44:52.845303 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:52.845254 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:53.364017 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:53.363954 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:53.366924 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:44:53.366890 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:44:53.419046 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:53.419018 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pmt9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:44:53.419165 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:53.419082 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" podUID="f7f198b8-0ce4-4ac4-9339-944e7f829bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:44:54.419251 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:54.419214 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pmt9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:44:54.419655 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:54.419267 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" podUID="f7f198b8-0ce4-4ac4-9339-944e7f829bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:44:55.418566 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:55.418523 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pmt9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:44:55.418767 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:55.418576 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" podUID="f7f198b8-0ce4-4ac4-9339-944e7f829bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:44:55.993420 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:55.993384 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:56.428020 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:56.427969 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:56.758128 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:56.758037 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:57.423505 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:57.423468 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pmt9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:44:57.423934 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:57.423579 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" podUID="f7f198b8-0ce4-4ac4-9339-944e7f829bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:57.424916 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:57.424892 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:57.425364 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:57.425344 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:44:57.426511 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:44:57.426463 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 19:45:00.268501 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.268470 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv"] Oct 30 19:45:00.272877 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.272859 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" Oct 30 19:45:00.315927 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.315904 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb"] Oct 30 19:45:00.319853 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.319836 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" Oct 30 19:45:00.385876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.385149 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8j7h\" (UniqueName: \"kubernetes.io/projected/aa08078f-0638-4bb1-96bf-3d9e202e6e5f-kube-api-access-k8j7h\") pod \"maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb\" (UID: \"aa08078f-0638-4bb1-96bf-3d9e202e6e5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" Oct 30 19:45:00.385876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.385291 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa08078f-0638-4bb1-96bf-3d9e202e6e5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb\" (UID: \"aa08078f-0638-4bb1-96bf-3d9e202e6e5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" Oct 30 19:45:00.385876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.385370 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnzlv\" (UniqueName: \"kubernetes.io/projected/ec561302-f4da-4d76-bd3f-75e3d0f9fbea-kube-api-access-nnzlv\") pod \"maas-api-key-cleanup-29889825-drrzv\" (UID: \"ec561302-f4da-4d76-bd3f-75e3d0f9fbea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" Oct 30 19:45:00.385876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.385463 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec561302-f4da-4d76-bd3f-75e3d0f9fbea-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-drrzv\" (UID: \"ec561302-f4da-4d76-bd3f-75e3d0f9fbea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" Oct 30 19:45:00.391913 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.391886 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv"] Oct 30 19:45:00.425121 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.425097 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\"" Oct 30 19:45:00.462027 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.462000 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb"] Oct 30 19:45:00.487967 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.487938 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec561302-f4da-4d76-bd3f-75e3d0f9fbea-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-drrzv\" (UID: \"ec561302-f4da-4d76-bd3f-75e3d0f9fbea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" Oct 30 19:45:00.488146 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.488115 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8j7h\" (UniqueName: \"kubernetes.io/projected/aa08078f-0638-4bb1-96bf-3d9e202e6e5f-kube-api-access-k8j7h\") pod \"maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb\" (UID: \"aa08078f-0638-4bb1-96bf-3d9e202e6e5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" Oct 30 19:45:00.488269 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.488255 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa08078f-0638-4bb1-96bf-3d9e202e6e5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb\" (UID: \"aa08078f-0638-4bb1-96bf-3d9e202e6e5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" Oct 30 19:45:00.488401 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.488378 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnzlv\" (UniqueName: \"kubernetes.io/projected/ec561302-f4da-4d76-bd3f-75e3d0f9fbea-kube-api-access-nnzlv\") pod \"maas-api-key-cleanup-29889825-drrzv\" (UID: \"ec561302-f4da-4d76-bd3f-75e3d0f9fbea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" Oct 30 19:45:00.488803 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.488779 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec561302-f4da-4d76-bd3f-75e3d0f9fbea-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-drrzv\" (UID: \"ec561302-f4da-4d76-bd3f-75e3d0f9fbea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" Oct 30 19:45:00.488918 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.488898 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa08078f-0638-4bb1-96bf-3d9e202e6e5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb\" (UID: \"aa08078f-0638-4bb1-96bf-3d9e202e6e5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" Oct 30 19:45:00.548059 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.548003 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnzlv\" (UniqueName: \"kubernetes.io/projected/ec561302-f4da-4d76-bd3f-75e3d0f9fbea-kube-api-access-nnzlv\") pod \"maas-api-key-cleanup-29889825-drrzv\" (UID: \"ec561302-f4da-4d76-bd3f-75e3d0f9fbea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" Oct 30 19:45:00.549301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.549284 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8j7h\" (UniqueName: \"kubernetes.io/projected/aa08078f-0638-4bb1-96bf-3d9e202e6e5f-kube-api-access-k8j7h\") pod \"maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb\" (UID: \"aa08078f-0638-4bb1-96bf-3d9e202e6e5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" Oct 30 19:45:00.586690 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.586664 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" Oct 30 19:45:00.638609 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.638574 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" Oct 30 19:45:00.984849 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:00.984819 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv"] Oct 30 19:45:00.985209 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:45:00.985185 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec561302_f4da_4d76_bd3f_75e3d0f9fbea.slice/crio-310e68574c7347ffed2b548ec409b85c391539816831f1eacd51829ae5c2dc99 WatchSource:0}: Error finding container 310e68574c7347ffed2b548ec409b85c391539816831f1eacd51829ae5c2dc99: Status 404 returned error can't find the container with id 310e68574c7347ffed2b548ec409b85c391539816831f1eacd51829ae5c2dc99 Oct 30 19:45:01.033333 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:01.033310 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb"] Oct 30 19:45:01.034579 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:45:01.034558 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa08078f_0638_4bb1_96bf_3d9e202e6e5f.slice/crio-dbdfe635257ed807089093b83ff9ac6b03a084c4df60b74cc3f6d3bb493474df WatchSource:0}: Error finding container dbdfe635257ed807089093b83ff9ac6b03a084c4df60b74cc3f6d3bb493474df: Status 404 returned error can't find the container with id dbdfe635257ed807089093b83ff9ac6b03a084c4df60b74cc3f6d3bb493474df Oct 30 19:45:01.376019 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:01.375601 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:01.695179 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:01.695096 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" event={"ID":"aa08078f-0638-4bb1-96bf-3d9e202e6e5f","Type":"ContainerStarted","Data":"dbdfe635257ed807089093b83ff9ac6b03a084c4df60b74cc3f6d3bb493474df"} Oct 30 19:45:01.696396 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:01.696364 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" event={"ID":"ec561302-f4da-4d76-bd3f-75e3d0f9fbea","Type":"ContainerStarted","Data":"310e68574c7347ffed2b548ec409b85c391539816831f1eacd51829ae5c2dc99"} Oct 30 19:45:05.725916 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:05.725885 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" event={"ID":"ec561302-f4da-4d76-bd3f-75e3d0f9fbea","Type":"ContainerStarted","Data":"569c6902b030b821b9726a11744074a09b5937386713014e140dcd6ff4d5d6f5"} Oct 30 19:45:05.727548 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:05.727522 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" event={"ID":"aa08078f-0638-4bb1-96bf-3d9e202e6e5f","Type":"ContainerStarted","Data":"8a98ca2b8fc4a9337206352437a0c1ad613f829c0aea22d79a31408ad692122a"} Oct 30 19:45:05.841748 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:05.841700 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" podStartSLOduration=1.8684226640000001 podStartE2EDuration="5.841685488s" totalImagesPullingTime="3.973262824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.840224075 +0000 UTC m=+1752.775011605" watchObservedRunningTime="2026-10-30 19:45:05.841685488 +0000 UTC m=+1752.776473011" Oct 30 19:45:05.960776 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:05.960693 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" podStartSLOduration=1.739529782 podStartE2EDuration="5.960679152s" totalImagesPullingTime="4.22114937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.959273176 +0000 UTC m=+1752.894060700" watchObservedRunningTime="2026-10-30 19:45:05.960679152 +0000 UTC m=+1752.895466675" Oct 30 19:45:06.364357 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:06.364325 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:06.734975 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:06.734890 4433 generic.go:356] "Generic (PLEG): container finished" podID="aa08078f-0638-4bb1-96bf-3d9e202e6e5f" containerID="8a98ca2b8fc4a9337206352437a0c1ad613f829c0aea22d79a31408ad692122a" exitCode=60 Oct 30 19:45:06.735425 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:06.734974 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" event={"ID":"aa08078f-0638-4bb1-96bf-3d9e202e6e5f","Type":"ContainerDied","Data":"8a98ca2b8fc4a9337206352437a0c1ad613f829c0aea22d79a31408ad692122a"} Oct 30 19:45:06.735425 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:06.735197 4433 scope.go:122] "RemoveContainer" containerID="8a98ca2b8fc4a9337206352437a0c1ad613f829c0aea22d79a31408ad692122a" Oct 30 19:45:06.736630 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:06.736607 4433 generic.go:356] "Generic (PLEG): container finished" podID="ec561302-f4da-4d76-bd3f-75e3d0f9fbea" containerID="569c6902b030b821b9726a11744074a09b5937386713014e140dcd6ff4d5d6f5" exitCode=60 Oct 30 19:45:06.736716 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:06.736672 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" event={"ID":"ec561302-f4da-4d76-bd3f-75e3d0f9fbea","Type":"ContainerDied","Data":"569c6902b030b821b9726a11744074a09b5937386713014e140dcd6ff4d5d6f5"} Oct 30 19:45:06.736941 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:06.736924 4433 scope.go:122] "RemoveContainer" containerID="569c6902b030b821b9726a11744074a09b5937386713014e140dcd6ff4d5d6f5" Oct 30 19:45:07.744495 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:07.744463 4433 generic.go:356] "Generic (PLEG): container finished" podID="aa08078f-0638-4bb1-96bf-3d9e202e6e5f" containerID="7e11ece2011da70adc67a419f161b20af3cb51b35943e75bca379b11d38aab32" exitCode=60 Oct 30 19:45:07.744880 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:07.744540 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" event={"ID":"aa08078f-0638-4bb1-96bf-3d9e202e6e5f","Type":"ContainerDied","Data":"7e11ece2011da70adc67a419f161b20af3cb51b35943e75bca379b11d38aab32"} Oct 30 19:45:07.744880 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:07.744613 4433 scope.go:122] "RemoveContainer" containerID="8a98ca2b8fc4a9337206352437a0c1ad613f829c0aea22d79a31408ad692122a" Oct 30 19:45:07.744880 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:07.744845 4433 scope.go:122] "RemoveContainer" containerID="7e11ece2011da70adc67a419f161b20af3cb51b35943e75bca379b11d38aab32" Oct 30 19:45:07.745167 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:45:07.745146 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb_odh-ai-gateway-infra(aa08078f-0638-4bb1-96bf-3d9e202e6e5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" podUID="aa08078f-0638-4bb1-96bf-3d9e202e6e5f" Oct 30 19:45:07.746717 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:07.746698 4433 generic.go:356] "Generic (PLEG): container finished" podID="ec561302-f4da-4d76-bd3f-75e3d0f9fbea" containerID="d9cc31107406e092f31636907600152ee537f21bbdd7db11f0e5deb332d70a71" exitCode=60 Oct 30 19:45:07.746804 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:07.746775 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" event={"ID":"ec561302-f4da-4d76-bd3f-75e3d0f9fbea","Type":"ContainerDied","Data":"d9cc31107406e092f31636907600152ee537f21bbdd7db11f0e5deb332d70a71"} Oct 30 19:45:07.746969 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:07.746954 4433 scope.go:122] "RemoveContainer" containerID="d9cc31107406e092f31636907600152ee537f21bbdd7db11f0e5deb332d70a71" Oct 30 19:45:07.747245 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:45:07.747229 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-drrzv_odh-ai-gateway-infra(ec561302-f4da-4d76-bd3f-75e3d0f9fbea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" podUID="ec561302-f4da-4d76-bd3f-75e3d0f9fbea" Oct 30 19:45:07.756423 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:07.756406 4433 scope.go:122] "RemoveContainer" containerID="569c6902b030b821b9726a11744074a09b5937386713014e140dcd6ff4d5d6f5" Oct 30 19:45:08.755212 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:08.755185 4433 scope.go:122] "RemoveContainer" containerID="7e11ece2011da70adc67a419f161b20af3cb51b35943e75bca379b11d38aab32" Oct 30 19:45:08.755607 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:45:08.755416 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb_odh-ai-gateway-infra(aa08078f-0638-4bb1-96bf-3d9e202e6e5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" podUID="aa08078f-0638-4bb1-96bf-3d9e202e6e5f" Oct 30 19:45:08.756922 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:08.756904 4433 scope.go:122] "RemoveContainer" containerID="d9cc31107406e092f31636907600152ee537f21bbdd7db11f0e5deb332d70a71" Oct 30 19:45:08.757160 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:45:08.757141 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-drrzv_odh-ai-gateway-infra(ec561302-f4da-4d76-bd3f-75e3d0f9fbea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" podUID="ec561302-f4da-4d76-bd3f-75e3d0f9fbea" Oct 30 19:45:10.853468 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:10.853435 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:11.132914 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:11.132880 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:15.122344 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:15.122312 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:15.849032 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:15.848918 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:16.202059 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:16.200956 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:16.538336 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:16.538251 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:18.060050 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.060014 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr"] Oct 30 19:45:18.064405 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.064385 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.119402 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.119372 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-94581ab2-openshift-default-dockercfg-l78tx\"" Oct 30 19:45:18.193245 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.193210 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr"] Oct 30 19:45:18.207103 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.207078 4433 scope.go:122] "RemoveContainer" containerID="7e11ece2011da70adc67a419f161b20af3cb51b35943e75bca379b11d38aab32" Oct 30 19:45:18.217380 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.217354 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-credential-socket\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.217476 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.217396 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istiod-ca-cert\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.217549 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.217469 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-podinfo\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.217549 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.217537 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-token\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.217667 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.217582 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-workload-socket\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.217667 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.217613 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-data\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.217790 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.217703 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-workload-certs\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.217790 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.217746 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9phh\" (UniqueName: \"kubernetes.io/projected/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-kube-api-access-j9phh\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.217897 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.217834 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-envoy\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.318889 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.318861 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-workload-certs\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.319062 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.319002 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9phh\" (UniqueName: \"kubernetes.io/projected/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-kube-api-access-j9phh\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.319141 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.319114 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-envoy\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.319283 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.319259 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-credential-socket\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.319353 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.319293 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-workload-certs\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.319411 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.319370 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istiod-ca-cert\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.319466 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.319432 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-podinfo\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.319824 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.319530 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-token\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.319824 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.319605 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-workload-socket\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.319824 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.319667 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-credential-socket\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.319824 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.319681 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-data\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.320141 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.319882 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-workload-socket\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.320141 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.320005 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istiod-ca-cert\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.320236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.320168 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-data\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.321766 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.321744 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-podinfo\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.322055 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.322032 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-envoy\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.378324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.378289 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-token\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.378581 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.378554 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9phh\" (UniqueName: \"kubernetes.io/projected/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-kube-api-access-j9phh\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.379197 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.379174 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:18.874270 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.874244 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr"] Oct 30 19:45:18.875476 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:45:18.875454 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a2f39d_cdaf_4fb3_9eb0_e54895ebc217.slice/crio-9a85ae49792f7cdf93f93eca93e56e4e322851ec14b9ca72637b3fc414f861a2 WatchSource:0}: Error finding container 9a85ae49792f7cdf93f93eca93e56e4e322851ec14b9ca72637b3fc414f861a2: Status 404 returned error can't find the container with id 9a85ae49792f7cdf93f93eca93e56e4e322851ec14b9ca72637b3fc414f861a2 Oct 30 19:45:18.877917 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.877885 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:18.877993 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.877951 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:18.878036 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.878017 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:18.888330 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.888302 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" event={"ID":"aa08078f-0638-4bb1-96bf-3d9e202e6e5f","Type":"ContainerStarted","Data":"d3a09e8558841fd04c9c17a991df698a1ff03cd889a036148bd731f08aaff495"} Oct 30 19:45:18.889968 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:18.889940 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" event={"ID":"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217","Type":"ContainerStarted","Data":"9a85ae49792f7cdf93f93eca93e56e4e322851ec14b9ca72637b3fc414f861a2"} Oct 30 19:45:19.908001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:19.907474 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" event={"ID":"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217","Type":"ContainerStarted","Data":"692c6a3c6d9ab12ec999a8091d5e0251cc74117ca3970dcc4710a9a612add8d4"} Oct 30 19:45:19.916998 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:19.909807 4433 generic.go:356] "Generic (PLEG): container finished" podID="aa08078f-0638-4bb1-96bf-3d9e202e6e5f" containerID="d3a09e8558841fd04c9c17a991df698a1ff03cd889a036148bd731f08aaff495" exitCode=60 Oct 30 19:45:19.916998 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:19.909849 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" event={"ID":"aa08078f-0638-4bb1-96bf-3d9e202e6e5f","Type":"ContainerDied","Data":"d3a09e8558841fd04c9c17a991df698a1ff03cd889a036148bd731f08aaff495"} Oct 30 19:45:19.916998 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:19.909911 4433 scope.go:122] "RemoveContainer" containerID="7e11ece2011da70adc67a419f161b20af3cb51b35943e75bca379b11d38aab32" Oct 30 19:45:19.916998 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:19.910404 4433 scope.go:122] "RemoveContainer" containerID="d3a09e8558841fd04c9c17a991df698a1ff03cd889a036148bd731f08aaff495" Oct 30 19:45:19.916998 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:45:19.910773 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb_odh-ai-gateway-infra(aa08078f-0638-4bb1-96bf-3d9e202e6e5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" podUID="aa08078f-0638-4bb1-96bf-3d9e202e6e5f" Oct 30 19:45:20.041519 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:20.041479 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb"] Oct 30 19:45:20.080430 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:20.080385 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:20.120815 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:20.120745 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" podStartSLOduration=2.120725648 podStartE2EDuration="2.120725648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:20.078702284 +0000 UTC m=+1767.013489808" watchObservedRunningTime="2026-10-30 19:45:20.120725648 +0000 UTC m=+1767.055513172" Oct 30 19:45:20.380566 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:20.380521 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:20.381640 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:20.381612 4433 patch_prober.go:29] interesting pod/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:45:20.381773 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:20.381655 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" podUID="97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 19:45:20.417295 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:20.417250 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:21.158379 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:21.158342 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" Oct 30 19:45:21.243656 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:21.243576 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:21.264167 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:21.264137 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa08078f-0638-4bb1-96bf-3d9e202e6e5f-kube-api-access-k8j7h\" (UniqueName: \"kubernetes.io/projected/aa08078f-0638-4bb1-96bf-3d9e202e6e5f-kube-api-access-k8j7h\") pod \"aa08078f-0638-4bb1-96bf-3d9e202e6e5f\" (UID: \"aa08078f-0638-4bb1-96bf-3d9e202e6e5f\") " Oct 30 19:45:21.264306 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:21.264174 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa08078f-0638-4bb1-96bf-3d9e202e6e5f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa08078f-0638-4bb1-96bf-3d9e202e6e5f-maas-api-service-ca\") pod \"aa08078f-0638-4bb1-96bf-3d9e202e6e5f\" (UID: \"aa08078f-0638-4bb1-96bf-3d9e202e6e5f\") " Oct 30 19:45:21.264550 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:21.264529 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa08078f-0638-4bb1-96bf-3d9e202e6e5f-maas-api-service-ca" pod "aa08078f-0638-4bb1-96bf-3d9e202e6e5f" (UID: "aa08078f-0638-4bb1-96bf-3d9e202e6e5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:21.266381 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:21.266353 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa08078f-0638-4bb1-96bf-3d9e202e6e5f-kube-api-access-k8j7h" pod "aa08078f-0638-4bb1-96bf-3d9e202e6e5f" (UID: "aa08078f-0638-4bb1-96bf-3d9e202e6e5f"). InnerVolumeSpecName "kube-api-access-k8j7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:21.366502 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:21.366464 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8j7h\" (UniqueName: \"kubernetes.io/projected/aa08078f-0638-4bb1-96bf-3d9e202e6e5f-kube-api-access-k8j7h\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:45:21.366502 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:21.366497 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa08078f-0638-4bb1-96bf-3d9e202e6e5f-maas-api-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:45:21.380039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:21.379974 4433 patch_prober.go:29] interesting pod/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:45:21.380177 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:21.380064 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" podUID="97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 19:45:21.935736 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:21.935702 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" Oct 30 19:45:21.935940 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:21.935731 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb" event={"ID":"aa08078f-0638-4bb1-96bf-3d9e202e6e5f","Type":"ContainerDied","Data":"dbdfe635257ed807089093b83ff9ac6b03a084c4df60b74cc3f6d3bb493474df"} Oct 30 19:45:21.935940 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:21.935800 4433 scope.go:122] "RemoveContainer" containerID="d3a09e8558841fd04c9c17a991df698a1ff03cd889a036148bd731f08aaff495" Oct 30 19:45:22.120440 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:22.120404 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb"] Oct 30 19:45:22.167516 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:22.167490 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-686689-29889825-7f5fb"] Oct 30 19:45:22.215223 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:22.214470 4433 scope.go:122] "RemoveContainer" containerID="d9cc31107406e092f31636907600152ee537f21bbdd7db11f0e5deb332d70a71" Oct 30 19:45:22.380656 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:22.380623 4433 patch_prober.go:29] interesting pod/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:45:22.380777 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:22.380698 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" podUID="97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 19:45:22.944974 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:22.944940 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" event={"ID":"ec561302-f4da-4d76-bd3f-75e3d0f9fbea","Type":"ContainerStarted","Data":"282300174f79d15a505b5aac60bbbf450d9bac37a4e55fdfd102dd910cfe82f8"} Oct 30 19:45:22.946602 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:22.946575 4433 generic.go:356] "Generic (PLEG): container finished" podID="44e31a3a-35aa-49a0-afd1-f722ecca6c08" containerID="4c3a9f2eaef934539ec36dc115f641b3fe8c8e098a2c7f3331db556f2678c221" exitCode=0 Oct 30 19:45:22.946732 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:22.946608 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" event={"ID":"44e31a3a-35aa-49a0-afd1-f722ecca6c08","Type":"ContainerDied","Data":"4c3a9f2eaef934539ec36dc115f641b3fe8c8e098a2c7f3331db556f2678c221"} Oct 30 19:45:22.947058 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:22.947040 4433 scope.go:122] "RemoveContainer" containerID="4c3a9f2eaef934539ec36dc115f641b3fe8c8e098a2c7f3331db556f2678c221" Oct 30 19:45:22.948591 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:22.948564 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:22.948680 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:22.948638 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:22.948680 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:22.948674 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:23.233166 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:23.233084 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa08078f-0638-4bb1-96bf-3d9e202e6e5f" path="/var/lib/kubelet/pods/aa08078f-0638-4bb1-96bf-3d9e202e6e5f/volumes" Oct 30 19:45:23.367570 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:45:23.367534 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:45:23.380634 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:23.380602 4433 patch_prober.go:29] interesting pod/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:45:23.380757 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:23.380660 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" podUID="97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 19:45:23.583392 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:23.583347 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:23.866003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:23.862687 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:24.370377 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.370336 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv"] Oct 30 19:45:24.380671 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.380371 4433 patch_prober.go:29] interesting pod/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:45:24.381145 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.381112 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" podUID="97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 19:45:24.438029 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.420137 4433 generic.go:356] "Generic (PLEG): container finished" podID="ec561302-f4da-4d76-bd3f-75e3d0f9fbea" containerID="282300174f79d15a505b5aac60bbbf450d9bac37a4e55fdfd102dd910cfe82f8" exitCode=60 Oct 30 19:45:24.438029 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.420259 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" event={"ID":"ec561302-f4da-4d76-bd3f-75e3d0f9fbea","Type":"ContainerDied","Data":"282300174f79d15a505b5aac60bbbf450d9bac37a4e55fdfd102dd910cfe82f8"} Oct 30 19:45:24.438029 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.420316 4433 scope.go:122] "RemoveContainer" containerID="d9cc31107406e092f31636907600152ee537f21bbdd7db11f0e5deb332d70a71" Oct 30 19:45:24.446817 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.446782 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" event={"ID":"44e31a3a-35aa-49a0-afd1-f722ecca6c08","Type":"ContainerStarted","Data":"53fda437b1d6cec9e0b1834d744907a9933a028ff7dd69d67d5ee78b3e5e9be0"} Oct 30 19:45:24.641126 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.634390 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:45:24.641126 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.634580 4433 patch_prober.go:29] interesting pod/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:45:24.641126 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.634630 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" podUID="44e31a3a-35aa-49a0-afd1-f722ecca6c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:45:24.751504 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.751476 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" Oct 30 19:45:24.776997 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.776952 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec561302-f4da-4d76-bd3f-75e3d0f9fbea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec561302-f4da-4d76-bd3f-75e3d0f9fbea-maas-api-service-ca\") pod \"ec561302-f4da-4d76-bd3f-75e3d0f9fbea\" (UID: \"ec561302-f4da-4d76-bd3f-75e3d0f9fbea\") " Oct 30 19:45:24.777224 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.777185 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec561302-f4da-4d76-bd3f-75e3d0f9fbea-kube-api-access-nnzlv\" (UniqueName: \"kubernetes.io/projected/ec561302-f4da-4d76-bd3f-75e3d0f9fbea-kube-api-access-nnzlv\") pod \"ec561302-f4da-4d76-bd3f-75e3d0f9fbea\" (UID: \"ec561302-f4da-4d76-bd3f-75e3d0f9fbea\") " Oct 30 19:45:24.777327 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.777303 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec561302-f4da-4d76-bd3f-75e3d0f9fbea-maas-api-service-ca" pod "ec561302-f4da-4d76-bd3f-75e3d0f9fbea" (UID: "ec561302-f4da-4d76-bd3f-75e3d0f9fbea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:24.777817 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.777797 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec561302-f4da-4d76-bd3f-75e3d0f9fbea-maas-api-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:45:24.779309 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.779291 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec561302-f4da-4d76-bd3f-75e3d0f9fbea-kube-api-access-nnzlv" pod "ec561302-f4da-4d76-bd3f-75e3d0f9fbea" (UID: "ec561302-f4da-4d76-bd3f-75e3d0f9fbea"). InnerVolumeSpecName "kube-api-access-nnzlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:24.880466 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:24.880425 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnzlv\" (UniqueName: \"kubernetes.io/projected/ec561302-f4da-4d76-bd3f-75e3d0f9fbea-kube-api-access-nnzlv\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:45:25.383538 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:25.383507 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:25.384020 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:25.383839 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:25.390198 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:25.390176 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:45:25.456053 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:25.456023 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" Oct 30 19:45:25.456053 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:25.456044 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv" event={"ID":"ec561302-f4da-4d76-bd3f-75e3d0f9fbea","Type":"ContainerDied","Data":"310e68574c7347ffed2b548ec409b85c391539816831f1eacd51829ae5c2dc99"} Oct 30 19:45:25.456231 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:25.456099 4433 scope.go:122] "RemoveContainer" containerID="282300174f79d15a505b5aac60bbbf450d9bac37a4e55fdfd102dd910cfe82f8" Oct 30 19:45:25.614971 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:25.614938 4433 patch_prober.go:29] interesting pod/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:45:25.615150 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:25.615016 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" podUID="44e31a3a-35aa-49a0-afd1-f722ecca6c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:45:25.620756 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:25.620722 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv"] Oct 30 19:45:25.624455 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:25.624424 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-drrzv"] Oct 30 19:45:26.417898 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:26.417853 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:26.614801 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:26.614771 4433 patch_prober.go:29] interesting pod/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:45:26.614963 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:26.614828 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" podUID="44e31a3a-35aa-49a0-afd1-f722ecca6c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:45:27.217283 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:27.217249 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec561302-f4da-4d76-bd3f-75e3d0f9fbea" path="/var/lib/kubelet/pods/ec561302-f4da-4d76-bd3f-75e3d0f9fbea/volumes" Oct 30 19:45:27.614672 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:27.614632 4433 patch_prober.go:29] interesting pod/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:45:27.615078 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:27.614699 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" podUID="44e31a3a-35aa-49a0-afd1-f722ecca6c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:45:28.614996 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:28.614948 4433 patch_prober.go:29] interesting pod/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:45:28.615503 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:28.615046 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" podUID="44e31a3a-35aa-49a0-afd1-f722ecca6c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:45:29.619706 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:29.619676 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:45:29.620183 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:29.620160 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:45:29.620900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:29.620875 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:45:29.804171 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:29.804144 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:30.066353 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:30.066324 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:31.237722 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:31.237668 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:36.425657 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:36.425620 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:38.849738 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:38.849703 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:39.128436 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:39.128354 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:40.768191 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:40.768155 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:41.510797 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:41.510762 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:44.990884 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:44.990850 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz"] Oct 30 19:45:45.363015 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:45.362936 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:45.605894 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:45.605858 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:46.401826 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:46.401793 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:46.648755 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:46.648720 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:47.025294 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:47.025260 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:49.991863 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:49.991824 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" podUID="90239164-408c-440e-9b1d-d2cfc6745937" containerName="payload-pre-processing" containerID="cri-o://923d651c09d01e9dabc7c685086de5f9b20d625f0b34ff481acfd84a0f384b4f" gracePeriod=25 Oct 30 19:45:50.260720 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:50.260661 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" Oct 30 19:45:50.349104 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:50.348843 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90239164-408c-440e-9b1d-d2cfc6745937-kube-api-access-gcgjf\" (UniqueName: \"kubernetes.io/projected/90239164-408c-440e-9b1d-d2cfc6745937-kube-api-access-gcgjf\") pod \"90239164-408c-440e-9b1d-d2cfc6745937\" (UID: \"90239164-408c-440e-9b1d-d2cfc6745937\") " Oct 30 19:45:50.349894 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:50.349562 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90239164-408c-440e-9b1d-d2cfc6745937-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90239164-408c-440e-9b1d-d2cfc6745937-plugins-config-volume\") pod \"90239164-408c-440e-9b1d-d2cfc6745937\" (UID: \"90239164-408c-440e-9b1d-d2cfc6745937\") " Oct 30 19:45:50.352133 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:50.352086 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90239164-408c-440e-9b1d-d2cfc6745937-plugins-config-volume" pod "90239164-408c-440e-9b1d-d2cfc6745937" (UID: "90239164-408c-440e-9b1d-d2cfc6745937"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:50.354446 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:50.354419 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90239164-408c-440e-9b1d-d2cfc6745937-kube-api-access-gcgjf" pod "90239164-408c-440e-9b1d-d2cfc6745937" (UID: "90239164-408c-440e-9b1d-d2cfc6745937"). InnerVolumeSpecName "kube-api-access-gcgjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:50.451668 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:50.451639 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcgjf\" (UniqueName: \"kubernetes.io/projected/90239164-408c-440e-9b1d-d2cfc6745937-kube-api-access-gcgjf\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:45:50.451668 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:50.451667 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90239164-408c-440e-9b1d-d2cfc6745937-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:45:50.630971 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:50.630942 4433 generic.go:356] "Generic (PLEG): container finished" podID="90239164-408c-440e-9b1d-d2cfc6745937" containerID="923d651c09d01e9dabc7c685086de5f9b20d625f0b34ff481acfd84a0f384b4f" exitCode=0 Oct 30 19:45:50.631147 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:50.631018 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" Oct 30 19:45:50.631147 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:50.631016 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" event={"ID":"90239164-408c-440e-9b1d-d2cfc6745937","Type":"ContainerDied","Data":"923d651c09d01e9dabc7c685086de5f9b20d625f0b34ff481acfd84a0f384b4f"} Oct 30 19:45:50.631147 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:50.631131 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz" event={"ID":"90239164-408c-440e-9b1d-d2cfc6745937","Type":"ContainerDied","Data":"e1a94fe644165018ea6f04062d6414c9df813920c4db2c875a93d254b0073210"} Oct 30 19:45:50.631272 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:50.631170 4433 scope.go:122] "RemoveContainer" containerID="923d651c09d01e9dabc7c685086de5f9b20d625f0b34ff481acfd84a0f384b4f" Oct 30 19:45:50.641368 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:50.641349 4433 scope.go:122] "RemoveContainer" containerID="923d651c09d01e9dabc7c685086de5f9b20d625f0b34ff481acfd84a0f384b4f" Oct 30 19:45:50.641627 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:45:50.641605 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"923d651c09d01e9dabc7c685086de5f9b20d625f0b34ff481acfd84a0f384b4f\": container with ID starting with 923d651c09d01e9dabc7c685086de5f9b20d625f0b34ff481acfd84a0f384b4f not found: ID does not exist" containerID="923d651c09d01e9dabc7c685086de5f9b20d625f0b34ff481acfd84a0f384b4f" Oct 30 19:45:50.641673 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:50.641636 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"923d651c09d01e9dabc7c685086de5f9b20d625f0b34ff481acfd84a0f384b4f"} err="failed to get container status \"923d651c09d01e9dabc7c685086de5f9b20d625f0b34ff481acfd84a0f384b4f\": rpc error: code = NotFound desc = could not find container \"923d651c09d01e9dabc7c685086de5f9b20d625f0b34ff481acfd84a0f384b4f\": container with ID starting with 923d651c09d01e9dabc7c685086de5f9b20d625f0b34ff481acfd84a0f384b4f not found: ID does not exist" Oct 30 19:45:50.805993 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:50.805948 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz"] Oct 30 19:45:50.827858 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:50.827832 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-94581ab2-6fd5988447-5vjxz"] Oct 30 19:45:51.215252 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:51.215224 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90239164-408c-440e-9b1d-d2cfc6745937" path="/var/lib/kubelet/pods/90239164-408c-440e-9b1d-d2cfc6745937/volumes" Oct 30 19:45:51.407517 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:51.407474 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:53.369146 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:45:53.369112 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:45:53.508178 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:53.508145 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:53.807139 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:53.807057 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:54.058810 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:54.058724 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:55.341286 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:55.341245 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:56.214806 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:45:56.214769 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:00.869838 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:00.869798 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:01.557577 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:01.557548 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:01.818217 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:01.818190 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:02.801703 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:02.801673 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:05.539198 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:05.539162 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:09.875509 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:09.875474 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:10.195854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:10.195766 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:14.835633 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:14.835597 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:15.065960 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:15.065925 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:16.445742 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:16.445709 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:19.694500 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:19.694469 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:23.365260 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:46:23.365226 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:46:23.736739 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:23.736664 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:25.286711 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:25.286678 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:25.620755 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:25.620725 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:25.801223 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:25.801192 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr"] Oct 30 19:46:25.801408 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:25.801383 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" podUID="97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" containerName="istio-proxy" containerID="cri-o://692c6a3c6d9ab12ec999a8091d5e0251cc74117ca3970dcc4710a9a612add8d4" gracePeriod=30 Oct 30 19:46:25.856896 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:25.856865 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:26.089763 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:26.089732 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:26.546319 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:26.546198 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:27.190252 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:27.190217 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:27.446551 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:27.446438 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:30.082663 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:30.082581 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:30.733571 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:30.733533 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:31.004425 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.004360 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:31.089720 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.089696 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:46:31.119105 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.119076 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-envoy\") pod \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " Oct 30 19:46:31.119221 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.119134 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-podinfo\") pod \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " Oct 30 19:46:31.119221 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.119181 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istiod-ca-cert\") pod \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " Oct 30 19:46:31.119352 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.119338 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-workload-socket\") pod \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " Oct 30 19:46:31.119433 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.119399 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-kube-api-access-j9phh\" (UniqueName: \"kubernetes.io/projected/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-kube-api-access-j9phh\") pod \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " Oct 30 19:46:31.119491 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.119457 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-token\" (UniqueName: \"kubernetes.io/projected/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-token\") pod \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " Oct 30 19:46:31.119556 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.119535 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-workload-certs\") pod \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " Oct 30 19:46:31.119608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.119566 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-workload-socket" pod "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" (UID: "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:31.119608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.119579 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istiod-ca-cert" pod "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" (UID: "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:31.119717 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.119639 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-data\") pod \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " Oct 30 19:46:31.119717 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.119690 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-credential-socket\") pod \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\" (UID: \"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217\") " Oct 30 19:46:31.119850 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.119829 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-workload-certs" pod "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" (UID: "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:31.120006 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.119903 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-data" pod "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" (UID: "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:31.120124 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.120097 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-credential-socket" pod "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" (UID: "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:31.120372 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.120354 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:31.120450 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.120373 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:31.120450 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.120382 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:31.120450 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.120391 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:31.120450 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.120401 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:31.121562 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.121537 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-podinfo" pod "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" (UID: "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:46:31.121660 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.121540 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-envoy" pod "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" (UID: "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:31.121735 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.121714 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-kube-api-access-j9phh" pod "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" (UID: "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217"). InnerVolumeSpecName "kube-api-access-j9phh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:31.121788 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.121749 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-token" pod "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" (UID: "97a2f39d-cdaf-4fb3-9eb0-e54895ebc217"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:31.223081 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.223048 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9phh\" (UniqueName: \"kubernetes.io/projected/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-kube-api-access-j9phh\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:31.223081 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.223080 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:31.223292 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.223094 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:31.223292 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.223109 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:31.689012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.688946 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:31.909553 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.909519 4433 generic.go:356] "Generic (PLEG): container finished" podID="97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" containerID="692c6a3c6d9ab12ec999a8091d5e0251cc74117ca3970dcc4710a9a612add8d4" exitCode=0 Oct 30 19:46:31.909735 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.909561 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" Oct 30 19:46:31.909735 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.909612 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" event={"ID":"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217","Type":"ContainerDied","Data":"692c6a3c6d9ab12ec999a8091d5e0251cc74117ca3970dcc4710a9a612add8d4"} Oct 30 19:46:31.909735 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.909659 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr" event={"ID":"97a2f39d-cdaf-4fb3-9eb0-e54895ebc217","Type":"ContainerDied","Data":"9a85ae49792f7cdf93f93eca93e56e4e322851ec14b9ca72637b3fc414f861a2"} Oct 30 19:46:31.909735 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.909680 4433 scope.go:122] "RemoveContainer" containerID="692c6a3c6d9ab12ec999a8091d5e0251cc74117ca3970dcc4710a9a612add8d4" Oct 30 19:46:31.920456 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.920274 4433 scope.go:122] "RemoveContainer" containerID="692c6a3c6d9ab12ec999a8091d5e0251cc74117ca3970dcc4710a9a612add8d4" Oct 30 19:46:31.920549 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:46:31.920528 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"692c6a3c6d9ab12ec999a8091d5e0251cc74117ca3970dcc4710a9a612add8d4\": container with ID starting with 692c6a3c6d9ab12ec999a8091d5e0251cc74117ca3970dcc4710a9a612add8d4 not found: ID does not exist" containerID="692c6a3c6d9ab12ec999a8091d5e0251cc74117ca3970dcc4710a9a612add8d4" Oct 30 19:46:31.920613 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.920557 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"692c6a3c6d9ab12ec999a8091d5e0251cc74117ca3970dcc4710a9a612add8d4"} err="failed to get container status \"692c6a3c6d9ab12ec999a8091d5e0251cc74117ca3970dcc4710a9a612add8d4\": rpc error: code = NotFound desc = could not find container \"692c6a3c6d9ab12ec999a8091d5e0251cc74117ca3970dcc4710a9a612add8d4\": container with ID starting with 692c6a3c6d9ab12ec999a8091d5e0251cc74117ca3970dcc4710a9a612add8d4 not found: ID does not exist" Oct 30 19:46:31.926457 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:31.926413 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:32.090433 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:32.090396 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr"] Oct 30 19:46:32.090912 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:32.090453 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-b4dgr"] Oct 30 19:46:32.180255 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:32.177433 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:32.181691 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:32.181663 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7"] Oct 30 19:46:32.181905 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:32.181882 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" podUID="44e31a3a-35aa-49a0-afd1-f722ecca6c08" containerName="istio-proxy" containerID="cri-o://53fda437b1d6cec9e0b1834d744907a9933a028ff7dd69d67d5ee78b3e5e9be0" gracePeriod=30 Oct 30 19:46:32.411505 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:32.411406 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:32.992451 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:32.992420 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:33.214497 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:33.214469 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" path="/var/lib/kubelet/pods/97a2f39d-cdaf-4fb3-9eb0-e54895ebc217/volumes" Oct 30 19:46:33.637614 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:33.637580 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:33.873649 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:33.873611 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:34.436019 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.431041 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:34.670336 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.670302 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:34.753704 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.753584 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg"] Oct 30 19:46:34.754289 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754267 4433 cpu_manager.go:414] "containerMap: removing container" podUID="ec561302-f4da-4d76-bd3f-75e3d0f9fbea" containerName="cleanup" Oct 30 19:46:34.754289 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754286 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec561302-f4da-4d76-bd3f-75e3d0f9fbea" containerName="cleanup" Oct 30 19:46:34.754467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754312 4433 cpu_manager.go:414] "containerMap: removing container" podUID="aa08078f-0638-4bb1-96bf-3d9e202e6e5f" containerName="cleanup" Oct 30 19:46:34.754467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754318 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa08078f-0638-4bb1-96bf-3d9e202e6e5f" containerName="cleanup" Oct 30 19:46:34.754467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754326 4433 cpu_manager.go:414] "containerMap: removing container" podUID="ec561302-f4da-4d76-bd3f-75e3d0f9fbea" containerName="cleanup" Oct 30 19:46:34.754467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754331 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec561302-f4da-4d76-bd3f-75e3d0f9fbea" containerName="cleanup" Oct 30 19:46:34.754467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754341 4433 cpu_manager.go:414] "containerMap: removing container" podUID="aa08078f-0638-4bb1-96bf-3d9e202e6e5f" containerName="cleanup" Oct 30 19:46:34.754467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754346 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa08078f-0638-4bb1-96bf-3d9e202e6e5f" containerName="cleanup" Oct 30 19:46:34.754467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754354 4433 cpu_manager.go:414] "containerMap: removing container" podUID="aa08078f-0638-4bb1-96bf-3d9e202e6e5f" containerName="cleanup" Oct 30 19:46:34.754467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754359 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa08078f-0638-4bb1-96bf-3d9e202e6e5f" containerName="cleanup" Oct 30 19:46:34.754467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754380 4433 cpu_manager.go:414] "containerMap: removing container" podUID="90239164-408c-440e-9b1d-d2cfc6745937" containerName="payload-pre-processing" Oct 30 19:46:34.754467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754385 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90239164-408c-440e-9b1d-d2cfc6745937" containerName="payload-pre-processing" Oct 30 19:46:34.754467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754392 4433 cpu_manager.go:414] "containerMap: removing container" podUID="97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" containerName="istio-proxy" Oct 30 19:46:34.754467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754397 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" containerName="istio-proxy" Oct 30 19:46:34.754848 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754492 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="90239164-408c-440e-9b1d-d2cfc6745937" containerName="payload-pre-processing" Oct 30 19:46:34.754848 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754502 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec561302-f4da-4d76-bd3f-75e3d0f9fbea" containerName="cleanup" Oct 30 19:46:34.754848 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754507 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a2f39d-cdaf-4fb3-9eb0-e54895ebc217" containerName="istio-proxy" Oct 30 19:46:34.754848 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754518 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa08078f-0638-4bb1-96bf-3d9e202e6e5f" containerName="cleanup" Oct 30 19:46:34.754848 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754524 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec561302-f4da-4d76-bd3f-75e3d0f9fbea" containerName="cleanup" Oct 30 19:46:34.754848 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754529 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa08078f-0638-4bb1-96bf-3d9e202e6e5f" containerName="cleanup" Oct 30 19:46:34.754848 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.754536 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa08078f-0638-4bb1-96bf-3d9e202e6e5f" containerName="cleanup" Oct 30 19:46:34.759492 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.759474 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" Oct 30 19:46:34.826053 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.825870 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg"] Oct 30 19:46:34.861648 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.861611 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-8a3c7851-dockercfg-m7nh9\"" Oct 30 19:46:34.861849 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.861686 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-8a3c7851\"" Oct 30 19:46:34.915223 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.915192 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9"] Oct 30 19:46:34.915810 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.915786 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104e9da9-96a0-426b-9363-d5885dd2fe63-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg\" (UID: \"104e9da9-96a0-426b-9363-d5885dd2fe63\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" Oct 30 19:46:34.915928 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.915833 4433 cpu_manager.go:414] "containerMap: removing container" podUID="ec561302-f4da-4d76-bd3f-75e3d0f9fbea" containerName="cleanup" Oct 30 19:46:34.915928 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.915850 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec561302-f4da-4d76-bd3f-75e3d0f9fbea" containerName="cleanup" Oct 30 19:46:34.915928 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.915904 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrpwb\" (UniqueName: \"kubernetes.io/projected/104e9da9-96a0-426b-9363-d5885dd2fe63-kube-api-access-qrpwb\") pod \"payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg\" (UID: \"104e9da9-96a0-426b-9363-d5885dd2fe63\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" Oct 30 19:46:34.916057 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.916037 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec561302-f4da-4d76-bd3f-75e3d0f9fbea" containerName="cleanup" Oct 30 19:46:34.919624 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.919594 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" Oct 30 19:46:34.969864 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:34.969826 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9"] Oct 30 19:46:35.017806 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.017703 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bljtj\" (UniqueName: \"kubernetes.io/projected/1b11cfae-71ff-4a16-96e3-e1fe6848eb46-kube-api-access-bljtj\") pod \"payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9\" (UID: \"1b11cfae-71ff-4a16-96e3-e1fe6848eb46\") " pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" Oct 30 19:46:35.017999 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.017800 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104e9da9-96a0-426b-9363-d5885dd2fe63-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg\" (UID: \"104e9da9-96a0-426b-9363-d5885dd2fe63\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" Oct 30 19:46:35.017999 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.017901 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrpwb\" (UniqueName: \"kubernetes.io/projected/104e9da9-96a0-426b-9363-d5885dd2fe63-kube-api-access-qrpwb\") pod \"payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg\" (UID: \"104e9da9-96a0-426b-9363-d5885dd2fe63\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" Oct 30 19:46:35.017999 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.017965 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b11cfae-71ff-4a16-96e3-e1fe6848eb46-plugins-config-volume\") pod \"payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9\" (UID: \"1b11cfae-71ff-4a16-96e3-e1fe6848eb46\") " pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" Oct 30 19:46:35.019778 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.019739 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104e9da9-96a0-426b-9363-d5885dd2fe63-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg\" (UID: \"104e9da9-96a0-426b-9363-d5885dd2fe63\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" Oct 30 19:46:35.084081 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.084037 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrpwb\" (UniqueName: \"kubernetes.io/projected/104e9da9-96a0-426b-9363-d5885dd2fe63-kube-api-access-qrpwb\") pod \"payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg\" (UID: \"104e9da9-96a0-426b-9363-d5885dd2fe63\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" Oct 30 19:46:35.138190 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.138148 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b11cfae-71ff-4a16-96e3-e1fe6848eb46-plugins-config-volume\") pod \"payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9\" (UID: \"1b11cfae-71ff-4a16-96e3-e1fe6848eb46\") " pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" Oct 30 19:46:35.138493 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.138418 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bljtj\" (UniqueName: \"kubernetes.io/projected/1b11cfae-71ff-4a16-96e3-e1fe6848eb46-kube-api-access-bljtj\") pod \"payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9\" (UID: \"1b11cfae-71ff-4a16-96e3-e1fe6848eb46\") " pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" Oct 30 19:46:35.139033 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.139005 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b11cfae-71ff-4a16-96e3-e1fe6848eb46-plugins-config-volume\") pod \"payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9\" (UID: \"1b11cfae-71ff-4a16-96e3-e1fe6848eb46\") " pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" Oct 30 19:46:35.204802 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.204768 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bljtj\" (UniqueName: \"kubernetes.io/projected/1b11cfae-71ff-4a16-96e3-e1fe6848eb46-kube-api-access-bljtj\") pod \"payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9\" (UID: \"1b11cfae-71ff-4a16-96e3-e1fe6848eb46\") " pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" Oct 30 19:46:35.238166 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.238131 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" Oct 30 19:46:35.375760 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.375724 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" Oct 30 19:46:35.620083 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.620058 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9"] Oct 30 19:46:35.621037 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:46:35.621011 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b11cfae_71ff_4a16_96e3_e1fe6848eb46.slice/crio-01f89ac4e15d688b2062303214f8701666a9a8a85e8c8a6ba5b1c89e09b0d28b WatchSource:0}: Error finding container 01f89ac4e15d688b2062303214f8701666a9a8a85e8c8a6ba5b1c89e09b0d28b: Status 404 returned error can't find the container with id 01f89ac4e15d688b2062303214f8701666a9a8a85e8c8a6ba5b1c89e09b0d28b Oct 30 19:46:35.766629 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.766608 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg"] Oct 30 19:46:35.767154 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:46:35.767126 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104e9da9_96a0_426b_9363_d5885dd2fe63.slice/crio-377a9675a9944c6ceaa7700b4decb982a4e89f22d2ac5e343b6b9b36170e724c WatchSource:0}: Error finding container 377a9675a9944c6ceaa7700b4decb982a4e89f22d2ac5e343b6b9b36170e724c: Status 404 returned error can't find the container with id 377a9675a9944c6ceaa7700b4decb982a4e89f22d2ac5e343b6b9b36170e724c Oct 30 19:46:35.947150 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.946956 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" event={"ID":"1b11cfae-71ff-4a16-96e3-e1fe6848eb46","Type":"ContainerStarted","Data":"1869633ca601aa746c3e34e99de4d1cfb3600e09ad0bd2149d1cb951531d9cdf"} Oct 30 19:46:35.947150 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.947063 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" event={"ID":"1b11cfae-71ff-4a16-96e3-e1fe6848eb46","Type":"ContainerStarted","Data":"01f89ac4e15d688b2062303214f8701666a9a8a85e8c8a6ba5b1c89e09b0d28b"} Oct 30 19:46:35.947406 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.947152 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" Oct 30 19:46:35.949510 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.949480 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" event={"ID":"104e9da9-96a0-426b-9363-d5885dd2fe63","Type":"ContainerStarted","Data":"f5ccd83f6b59ef9d589e0a48a046cc9eb2d910e833c1e998e6d47aaf417173f1"} Oct 30 19:46:35.949644 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.949532 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" Oct 30 19:46:35.949644 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:35.949554 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" event={"ID":"104e9da9-96a0-426b-9363-d5885dd2fe63","Type":"ContainerStarted","Data":"377a9675a9944c6ceaa7700b4decb982a4e89f22d2ac5e343b6b9b36170e724c"} Oct 30 19:46:36.178849 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:36.178788 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" podStartSLOduration=2.178768497 podStartE2EDuration="2.178768497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:36.17774055 +0000 UTC m=+1843.112528074" watchObservedRunningTime="2026-10-30 19:46:36.178768497 +0000 UTC m=+1843.113556021" Oct 30 19:46:36.181136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:36.181092 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" podStartSLOduration=2.181078186 podStartE2EDuration="2.181078186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:36.064004096 +0000 UTC m=+1842.998791621" watchObservedRunningTime="2026-10-30 19:46:36.181078186 +0000 UTC m=+1843.115865710" Oct 30 19:46:37.149507 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.149479 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:37.477651 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.477629 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:46:37.499178 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.499152 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-token\" (UniqueName: \"kubernetes.io/projected/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-token\") pod \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " Oct 30 19:46:37.499290 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.499231 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e31a3a-35aa-49a0-afd1-f722ecca6c08-kube-api-access-qphht\" (UniqueName: \"kubernetes.io/projected/44e31a3a-35aa-49a0-afd1-f722ecca6c08-kube-api-access-qphht\") pod \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " Oct 30 19:46:37.499352 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.499291 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-podinfo\") pod \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " Oct 30 19:46:37.499352 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.499312 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-credential-socket\") pod \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " Oct 30 19:46:37.499352 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.499346 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-workload-socket\") pod \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " Oct 30 19:46:37.499521 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.499395 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-envoy\") pod \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " Oct 30 19:46:37.499573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.499556 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-workload-certs\") pod \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " Oct 30 19:46:37.499628 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.499594 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-data\") pod \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " Oct 30 19:46:37.499692 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.499676 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istiod-ca-cert\") pod \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\" (UID: \"44e31a3a-35aa-49a0-afd1-f722ecca6c08\") " Oct 30 19:46:37.499749 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.499687 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-workload-socket" pod "44e31a3a-35aa-49a0-afd1-f722ecca6c08" (UID: "44e31a3a-35aa-49a0-afd1-f722ecca6c08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:37.499962 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.499932 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-credential-socket" pod "44e31a3a-35aa-49a0-afd1-f722ecca6c08" (UID: "44e31a3a-35aa-49a0-afd1-f722ecca6c08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:37.500058 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.500019 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istiod-ca-cert" pod "44e31a3a-35aa-49a0-afd1-f722ecca6c08" (UID: "44e31a3a-35aa-49a0-afd1-f722ecca6c08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:37.500200 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.500179 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-workload-certs" pod "44e31a3a-35aa-49a0-afd1-f722ecca6c08" (UID: "44e31a3a-35aa-49a0-afd1-f722ecca6c08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:37.500314 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.500294 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-data" pod "44e31a3a-35aa-49a0-afd1-f722ecca6c08" (UID: "44e31a3a-35aa-49a0-afd1-f722ecca6c08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:37.500583 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.500564 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:37.500675 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.500589 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:37.500675 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.500610 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:37.500675 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.500621 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:37.500675 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.500630 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:37.501813 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.501793 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e31a3a-35aa-49a0-afd1-f722ecca6c08-kube-api-access-qphht" pod "44e31a3a-35aa-49a0-afd1-f722ecca6c08" (UID: "44e31a3a-35aa-49a0-afd1-f722ecca6c08"). InnerVolumeSpecName "kube-api-access-qphht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:37.501813 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.501802 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-podinfo" pod "44e31a3a-35aa-49a0-afd1-f722ecca6c08" (UID: "44e31a3a-35aa-49a0-afd1-f722ecca6c08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:46:37.501970 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.501825 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-token" pod "44e31a3a-35aa-49a0-afd1-f722ecca6c08" (UID: "44e31a3a-35aa-49a0-afd1-f722ecca6c08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:37.502200 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.502186 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-envoy" pod "44e31a3a-35aa-49a0-afd1-f722ecca6c08" (UID: "44e31a3a-35aa-49a0-afd1-f722ecca6c08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:37.602406 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.602383 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:37.602406 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.602405 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qphht\" (UniqueName: \"kubernetes.io/projected/44e31a3a-35aa-49a0-afd1-f722ecca6c08-kube-api-access-qphht\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:37.602515 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.602414 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:37.602515 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.602421 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44e31a3a-35aa-49a0-afd1-f722ecca6c08-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:37.781357 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.781262 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:37.966543 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.966509 4433 generic.go:356] "Generic (PLEG): container finished" podID="44e31a3a-35aa-49a0-afd1-f722ecca6c08" containerID="53fda437b1d6cec9e0b1834d744907a9933a028ff7dd69d67d5ee78b3e5e9be0" exitCode=0 Oct 30 19:46:37.966675 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.966568 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" Oct 30 19:46:37.966675 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.966590 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" event={"ID":"44e31a3a-35aa-49a0-afd1-f722ecca6c08","Type":"ContainerDied","Data":"53fda437b1d6cec9e0b1834d744907a9933a028ff7dd69d67d5ee78b3e5e9be0"} Oct 30 19:46:37.966675 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.966642 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7" event={"ID":"44e31a3a-35aa-49a0-afd1-f722ecca6c08","Type":"ContainerDied","Data":"048fd1f650cfb52a81bc15b14b74c89e9f05b14ed38594ddf530472f3a90ec4d"} Oct 30 19:46:37.966675 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.966671 4433 scope.go:122] "RemoveContainer" containerID="53fda437b1d6cec9e0b1834d744907a9933a028ff7dd69d67d5ee78b3e5e9be0" Oct 30 19:46:37.978189 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.978169 4433 scope.go:122] "RemoveContainer" containerID="4c3a9f2eaef934539ec36dc115f641b3fe8c8e098a2c7f3331db556f2678c221" Oct 30 19:46:37.987626 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.987611 4433 scope.go:122] "RemoveContainer" containerID="53fda437b1d6cec9e0b1834d744907a9933a028ff7dd69d67d5ee78b3e5e9be0" Oct 30 19:46:37.987930 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:46:37.987903 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53fda437b1d6cec9e0b1834d744907a9933a028ff7dd69d67d5ee78b3e5e9be0\": container with ID starting with 53fda437b1d6cec9e0b1834d744907a9933a028ff7dd69d67d5ee78b3e5e9be0 not found: ID does not exist" containerID="53fda437b1d6cec9e0b1834d744907a9933a028ff7dd69d67d5ee78b3e5e9be0" Oct 30 19:46:37.988039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.987942 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fda437b1d6cec9e0b1834d744907a9933a028ff7dd69d67d5ee78b3e5e9be0"} err="failed to get container status \"53fda437b1d6cec9e0b1834d744907a9933a028ff7dd69d67d5ee78b3e5e9be0\": rpc error: code = NotFound desc = could not find container \"53fda437b1d6cec9e0b1834d744907a9933a028ff7dd69d67d5ee78b3e5e9be0\": container with ID starting with 53fda437b1d6cec9e0b1834d744907a9933a028ff7dd69d67d5ee78b3e5e9be0 not found: ID does not exist" Oct 30 19:46:37.988039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.987966 4433 scope.go:122] "RemoveContainer" containerID="4c3a9f2eaef934539ec36dc115f641b3fe8c8e098a2c7f3331db556f2678c221" Oct 30 19:46:37.988272 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:46:37.988246 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3a9f2eaef934539ec36dc115f641b3fe8c8e098a2c7f3331db556f2678c221\": container with ID starting with 4c3a9f2eaef934539ec36dc115f641b3fe8c8e098a2c7f3331db556f2678c221 not found: ID does not exist" containerID="4c3a9f2eaef934539ec36dc115f641b3fe8c8e098a2c7f3331db556f2678c221" Oct 30 19:46:37.988377 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:37.988281 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3a9f2eaef934539ec36dc115f641b3fe8c8e098a2c7f3331db556f2678c221"} err="failed to get container status \"4c3a9f2eaef934539ec36dc115f641b3fe8c8e098a2c7f3331db556f2678c221\": rpc error: code = NotFound desc = could not find container \"4c3a9f2eaef934539ec36dc115f641b3fe8c8e098a2c7f3331db556f2678c221\": container with ID starting with 4c3a9f2eaef934539ec36dc115f641b3fe8c8e098a2c7f3331db556f2678c221 not found: ID does not exist" Oct 30 19:46:38.023962 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:38.023939 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:38.135520 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:38.135499 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7"] Oct 30 19:46:38.184947 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:38.184922 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d74c1bd9-gw-openshift-default-7f6c7cf869-c8ls7"] Oct 30 19:46:39.214738 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:39.214702 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44e31a3a-35aa-49a0-afd1-f722ecca6c08" path="/var/lib/kubelet/pods/44e31a3a-35aa-49a0-afd1-f722ecca6c08/volumes" Oct 30 19:46:41.166264 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.166205 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:41.401687 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.401653 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:41.473571 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.473478 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp"] Oct 30 19:46:41.474527 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.474499 4433 cpu_manager.go:414] "containerMap: removing container" podUID="44e31a3a-35aa-49a0-afd1-f722ecca6c08" containerName="istio-proxy" Oct 30 19:46:41.474682 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.474666 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e31a3a-35aa-49a0-afd1-f722ecca6c08" containerName="istio-proxy" Oct 30 19:46:41.475225 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.475204 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e31a3a-35aa-49a0-afd1-f722ecca6c08" containerName="istio-proxy" Oct 30 19:46:41.475382 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.475355 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e31a3a-35aa-49a0-afd1-f722ecca6c08" containerName="istio-proxy" Oct 30 19:46:41.480930 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.480869 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" Oct 30 19:46:41.534425 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.534388 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp"] Oct 30 19:46:41.552865 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.552832 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009b4710-bd3c-4273-bbee-84ebfbe13896-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp\" (UID: \"009b4710-bd3c-4273-bbee-84ebfbe13896\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" Oct 30 19:46:41.553107 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.552996 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc7v8\" (UniqueName: \"kubernetes.io/projected/009b4710-bd3c-4273-bbee-84ebfbe13896-kube-api-access-vc7v8\") pod \"payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp\" (UID: \"009b4710-bd3c-4273-bbee-84ebfbe13896\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" Oct 30 19:46:41.578697 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.578194 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-df789477\"" Oct 30 19:46:41.578697 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.578394 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-df789477-dockercfg-h6mpz\"" Oct 30 19:46:41.630747 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.630717 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7"] Oct 30 19:46:41.631457 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.631440 4433 cpu_manager.go:414] "containerMap: removing container" podUID="44e31a3a-35aa-49a0-afd1-f722ecca6c08" containerName="istio-proxy" Oct 30 19:46:41.631532 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.631457 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e31a3a-35aa-49a0-afd1-f722ecca6c08" containerName="istio-proxy" Oct 30 19:46:41.635113 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.635089 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" Oct 30 19:46:41.654738 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.654711 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009b4710-bd3c-4273-bbee-84ebfbe13896-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp\" (UID: \"009b4710-bd3c-4273-bbee-84ebfbe13896\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" Oct 30 19:46:41.654839 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.654824 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc7v8\" (UniqueName: \"kubernetes.io/projected/009b4710-bd3c-4273-bbee-84ebfbe13896-kube-api-access-vc7v8\") pod \"payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp\" (UID: \"009b4710-bd3c-4273-bbee-84ebfbe13896\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" Oct 30 19:46:41.655558 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.655539 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009b4710-bd3c-4273-bbee-84ebfbe13896-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp\" (UID: \"009b4710-bd3c-4273-bbee-84ebfbe13896\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" Oct 30 19:46:41.697184 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.697149 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7"] Oct 30 19:46:41.715296 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.715259 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc7v8\" (UniqueName: \"kubernetes.io/projected/009b4710-bd3c-4273-bbee-84ebfbe13896-kube-api-access-vc7v8\") pod \"payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp\" (UID: \"009b4710-bd3c-4273-bbee-84ebfbe13896\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" Oct 30 19:46:41.756232 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.756139 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7fvc\" (UniqueName: \"kubernetes.io/projected/31b8ebe9-1b26-4870-b9ad-6535916b56f7-kube-api-access-n7fvc\") pod \"payload-processing-e2e-ait-df789477-58687686bb-nw8m7\" (UID: \"31b8ebe9-1b26-4870-b9ad-6535916b56f7\") " pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" Oct 30 19:46:41.756674 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.756647 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b8ebe9-1b26-4870-b9ad-6535916b56f7-plugins-config-volume\") pod \"payload-processing-e2e-ait-df789477-58687686bb-nw8m7\" (UID: \"31b8ebe9-1b26-4870-b9ad-6535916b56f7\") " pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" Oct 30 19:46:41.809010 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.807500 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" Oct 30 19:46:41.862297 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.859628 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b8ebe9-1b26-4870-b9ad-6535916b56f7-plugins-config-volume\") pod \"payload-processing-e2e-ait-df789477-58687686bb-nw8m7\" (UID: \"31b8ebe9-1b26-4870-b9ad-6535916b56f7\") " pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" Oct 30 19:46:41.862297 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.859801 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7fvc\" (UniqueName: \"kubernetes.io/projected/31b8ebe9-1b26-4870-b9ad-6535916b56f7-kube-api-access-n7fvc\") pod \"payload-processing-e2e-ait-df789477-58687686bb-nw8m7\" (UID: \"31b8ebe9-1b26-4870-b9ad-6535916b56f7\") " pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" Oct 30 19:46:41.862297 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.861036 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b8ebe9-1b26-4870-b9ad-6535916b56f7-plugins-config-volume\") pod \"payload-processing-e2e-ait-df789477-58687686bb-nw8m7\" (UID: \"31b8ebe9-1b26-4870-b9ad-6535916b56f7\") " pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" Oct 30 19:46:41.917684 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.917622 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7fvc\" (UniqueName: \"kubernetes.io/projected/31b8ebe9-1b26-4870-b9ad-6535916b56f7-kube-api-access-n7fvc\") pod \"payload-processing-e2e-ait-df789477-58687686bb-nw8m7\" (UID: \"31b8ebe9-1b26-4870-b9ad-6535916b56f7\") " pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" Oct 30 19:46:41.953309 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:41.953277 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" Oct 30 19:46:42.252998 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:42.252950 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp"] Oct 30 19:46:42.254959 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:46:42.254925 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009b4710_bd3c_4273_bbee_84ebfbe13896.slice/crio-f5e3eb92e04149c7097ad2b7f19a19b4d516c3fce64aa16d165bdf526e0c0a27 WatchSource:0}: Error finding container f5e3eb92e04149c7097ad2b7f19a19b4d516c3fce64aa16d165bdf526e0c0a27: Status 404 returned error can't find the container with id f5e3eb92e04149c7097ad2b7f19a19b4d516c3fce64aa16d165bdf526e0c0a27 Oct 30 19:46:42.369460 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:46:42.369412 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b8ebe9_1b26_4870_b9ad_6535916b56f7.slice/crio-bfbe2a827727461c396ff1efd27e6a53710903231220ffea5353cbc458134be5 WatchSource:0}: Error finding container bfbe2a827727461c396ff1efd27e6a53710903231220ffea5353cbc458134be5: Status 404 returned error can't find the container with id bfbe2a827727461c396ff1efd27e6a53710903231220ffea5353cbc458134be5 Oct 30 19:46:42.369585 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:42.369484 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7"] Oct 30 19:46:42.371341 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:42.371311 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:43.011587 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:43.011547 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" event={"ID":"31b8ebe9-1b26-4870-b9ad-6535916b56f7","Type":"ContainerStarted","Data":"9b0baeede1efc97f2c7c3af1e1183dee11ccc7faa1bd4e642ab895ff0f4b5a14"} Oct 30 19:46:43.011781 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:43.011751 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" Oct 30 19:46:43.011863 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:43.011799 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" event={"ID":"31b8ebe9-1b26-4870-b9ad-6535916b56f7","Type":"ContainerStarted","Data":"bfbe2a827727461c396ff1efd27e6a53710903231220ffea5353cbc458134be5"} Oct 30 19:46:43.013264 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:43.013224 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" event={"ID":"009b4710-bd3c-4273-bbee-84ebfbe13896","Type":"ContainerStarted","Data":"0a22cfe8e02833ea8b2cc4c52da3b725fd489bbb84a08dd75dec4469cdb193e0"} Oct 30 19:46:43.013374 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:43.013282 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" event={"ID":"009b4710-bd3c-4273-bbee-84ebfbe13896","Type":"ContainerStarted","Data":"f5e3eb92e04149c7097ad2b7f19a19b4d516c3fce64aa16d165bdf526e0c0a27"} Oct 30 19:46:43.013374 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:43.013312 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" Oct 30 19:46:43.176307 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:43.176231 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" podStartSLOduration=2.176213894 podStartE2EDuration="2.176213894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:43.125257276 +0000 UTC m=+1850.060044801" watchObservedRunningTime="2026-10-30 19:46:43.176213894 +0000 UTC m=+1850.111001419" Oct 30 19:46:43.244171 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:43.244118 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" podStartSLOduration=2.244104799 podStartE2EDuration="2.244104799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:43.23940708 +0000 UTC m=+1850.174194623" watchObservedRunningTime="2026-10-30 19:46:43.244104799 +0000 UTC m=+1850.178892323" Oct 30 19:46:45.373854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:45.373822 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:45.766835 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:45.766759 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:46.295860 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.295823 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth"] Oct 30 19:46:46.300470 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.300446 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.353671 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.352729 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-c75ed4-openshift-default-dockercfg-m2ws2\"" Oct 30 19:46:46.406283 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.406255 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth"] Oct 30 19:46:46.435780 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.435738 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-data\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.435954 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.435785 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-credential-socket\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.435954 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.435891 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-workload-certs\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.436114 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.436069 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-token\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.436225 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.436199 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-envoy\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.436302 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.436283 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-podinfo\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.436359 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.436330 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efa7df32-302f-44e6-aef2-c4aff86c2b78-istiod-ca-cert\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.436467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.436438 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gn8z\" (UniqueName: \"kubernetes.io/projected/efa7df32-302f-44e6-aef2-c4aff86c2b78-kube-api-access-4gn8z\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.436644 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.436626 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-workload-socket\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.494135 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.494107 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:46.538601 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.538554 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-token\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.538804 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.538635 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-envoy\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.538804 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.538671 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-podinfo\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.538804 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.538716 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efa7df32-302f-44e6-aef2-c4aff86c2b78-istiod-ca-cert\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.539011 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.538803 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gn8z\" (UniqueName: \"kubernetes.io/projected/efa7df32-302f-44e6-aef2-c4aff86c2b78-kube-api-access-4gn8z\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.539067 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.539006 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-workload-socket\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.539125 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.539062 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-data\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.539125 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.539099 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-credential-socket\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.539226 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.539144 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-workload-certs\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.539625 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.539599 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-workload-certs\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.540168 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.540147 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-workload-socket\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.540387 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.540368 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-credential-socket\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.540468 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.540430 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efa7df32-302f-44e6-aef2-c4aff86c2b78-istiod-ca-cert\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.540607 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.540580 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-data\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.551346 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.551257 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-envoy\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.581004 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.555348 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-podinfo\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.611316 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.597375 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-token\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.611316 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.598803 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gn8z\" (UniqueName: \"kubernetes.io/projected/efa7df32-302f-44e6-aef2-c4aff86c2b78-kube-api-access-4gn8z\") pod \"e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.619000 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.616267 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:46.786577 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.786492 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:46.960806 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.960777 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" Oct 30 19:46:46.961037 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:46.960965 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" Oct 30 19:46:47.075293 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:47.075265 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth"] Oct 30 19:46:47.076296 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:46:47.076268 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa7df32_302f_44e6_aef2_c4aff86c2b78.slice/crio-1421e57bda8ca83dc57b31c4eb3664a5cf63ee795d9be6b4aa79ffba06151560 WatchSource:0}: Error finding container 1421e57bda8ca83dc57b31c4eb3664a5cf63ee795d9be6b4aa79ffba06151560: Status 404 returned error can't find the container with id 1421e57bda8ca83dc57b31c4eb3664a5cf63ee795d9be6b4aa79ffba06151560 Oct 30 19:46:47.078879 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:47.078847 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:47.078951 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:47.078909 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:47.078951 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:47.078939 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:47.530910 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:47.530868 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:48.051308 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.051271 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" event={"ID":"efa7df32-302f-44e6-aef2-c4aff86c2b78","Type":"ContainerStarted","Data":"2841278e2d0b1dc47a4ca7dcea4fb66c1f7248b8418702608e13c991bd27ac41"} Oct 30 19:46:48.051500 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.051337 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" event={"ID":"efa7df32-302f-44e6-aef2-c4aff86c2b78","Type":"ContainerStarted","Data":"1421e57bda8ca83dc57b31c4eb3664a5cf63ee795d9be6b4aa79ffba06151560"} Oct 30 19:46:48.171547 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.171448 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" podStartSLOduration=2.171432202 podStartE2EDuration="2.171432202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:48.169525254 +0000 UTC m=+1855.104312778" watchObservedRunningTime="2026-10-30 19:46:48.171432202 +0000 UTC m=+1855.106219727" Oct 30 19:46:48.626003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.625239 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:48.628405 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.628376 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:48.857829 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.857790 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s"] Oct 30 19:46:48.862323 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.862301 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:48.966352 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.966273 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s"] Oct 30 19:46:48.977410 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.977370 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335bb767-624e-44ba-8dab-3bab19297c9e-istio-podinfo\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:48.977582 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.977423 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335bb767-624e-44ba-8dab-3bab19297c9e-istiod-ca-cert\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:48.977582 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.977490 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5snz\" (UniqueName: \"kubernetes.io/projected/335bb767-624e-44ba-8dab-3bab19297c9e-kube-api-access-w5snz\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:48.977582 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.977524 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-istio-envoy\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:48.977582 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.977554 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-credential-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:48.977795 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.977622 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-workload-certs\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:48.977795 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.977665 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-istio-data\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:48.977795 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.977754 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335bb767-624e-44ba-8dab-3bab19297c9e-istio-token\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:48.977921 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:48.977826 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-workload-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.008162 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.008049 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-cf9b6f-openshift-default-dockercfg-qcqbf\"" Oct 30 19:46:49.060230 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.060201 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:49.061229 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.061208 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:46:49.078954 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.078923 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5snz\" (UniqueName: \"kubernetes.io/projected/335bb767-624e-44ba-8dab-3bab19297c9e-kube-api-access-w5snz\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.079090 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.078973 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-istio-envoy\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.079090 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.079046 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-credential-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.079211 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.079107 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-workload-certs\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.079211 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.079151 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-istio-data\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.079301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.079223 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335bb767-624e-44ba-8dab-3bab19297c9e-istio-token\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.079301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.079282 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-workload-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.079469 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.079437 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335bb767-624e-44ba-8dab-3bab19297c9e-istio-podinfo\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.079469 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.079458 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-credential-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.079601 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.079473 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335bb767-624e-44ba-8dab-3bab19297c9e-istiod-ca-cert\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.079884 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.079863 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-workload-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.079957 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.079885 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-workload-certs\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.079957 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.079934 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-istio-data\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.080196 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.080174 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335bb767-624e-44ba-8dab-3bab19297c9e-istiod-ca-cert\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.081398 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.081381 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-istio-envoy\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.081892 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.081870 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335bb767-624e-44ba-8dab-3bab19297c9e-istio-podinfo\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.137001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.136757 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335bb767-624e-44ba-8dab-3bab19297c9e-istio-token\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.143814 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.143705 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5snz\" (UniqueName: \"kubernetes.io/projected/335bb767-624e-44ba-8dab-3bab19297c9e-kube-api-access-w5snz\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.176582 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.176557 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:49.750422 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.748672 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s"] Oct 30 19:46:49.754946 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.754894 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:49.755080 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.755047 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:49.755141 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:49.755117 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:50.069832 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:50.069787 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" event={"ID":"335bb767-624e-44ba-8dab-3bab19297c9e","Type":"ContainerStarted","Data":"851e8cd36984587096d208da4f9bd59200f65bd45d80bf281c1043c0060327f3"} Oct 30 19:46:50.070056 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:50.069854 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" event={"ID":"335bb767-624e-44ba-8dab-3bab19297c9e","Type":"ContainerStarted","Data":"fa59063059b1ddc348776f9e476f1ba71ae822230950cc77298c8d30c40c5e92"} Oct 30 19:46:50.177444 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:50.177411 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:50.178988 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:50.178948 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:46:50.179127 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:50.179011 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" podUID="335bb767-624e-44ba-8dab-3bab19297c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 19:46:50.187683 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:50.187623 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" podStartSLOduration=2.187603459 podStartE2EDuration="2.187603459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:50.185077612 +0000 UTC m=+1857.119865138" watchObservedRunningTime="2026-10-30 19:46:50.187603459 +0000 UTC m=+1857.122390985" Oct 30 19:46:51.177884 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:51.177848 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:46:51.178310 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:51.177909 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" podUID="335bb767-624e-44ba-8dab-3bab19297c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 19:46:52.177160 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:52.177129 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:46:52.177333 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:52.177179 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" podUID="335bb767-624e-44ba-8dab-3bab19297c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 19:46:52.803005 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:52.801708 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:53.176849 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:53.176812 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:46:53.177045 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:53.176865 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" podUID="335bb767-624e-44ba-8dab-3bab19297c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 19:46:53.367245 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:46:53.367202 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:46:54.025170 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:54.025126 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" Oct 30 19:46:54.026580 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:54.026555 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" Oct 30 19:46:54.177450 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:54.177416 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:46:54.177594 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:54.177475 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" podUID="335bb767-624e-44ba-8dab-3bab19297c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 19:46:55.177528 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:55.177491 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:46:55.177877 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:55.177539 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" podUID="335bb767-624e-44ba-8dab-3bab19297c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 19:46:56.181661 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:56.181631 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:56.182113 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:56.182092 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:56.182800 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:56.182781 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:46:57.647844 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:57.647806 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:59.354333 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:46:59.354259 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:00.222822 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:00.222786 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:01.016683 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:01.016643 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:01.277204 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:01.277131 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:02.165793 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:02.165752 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:02.684036 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:02.683999 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:03.840432 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:03.840401 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f"] Oct 30 19:47:03.846025 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:03.846008 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:03.992846 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:03.992818 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:03.993029 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:03.992934 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:03.993103 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:03.993063 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:03.993103 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:03.993098 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfgpk\" (UniqueName: \"kubernetes.io/projected/a9420be3-395c-4e5a-b565-001d9b8cf7f6-kube-api-access-rfgpk\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:03.993180 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:03.993151 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:03.993379 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:03.993353 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:03.993425 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:03.993392 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:03.993461 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:03.993438 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:03.993508 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:03.993466 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.052484 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.052449 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f"] Oct 30 19:47:04.095476 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.095405 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.095620 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.095505 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.095620 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.095592 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.095780 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.095631 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfgpk\" (UniqueName: \"kubernetes.io/projected/a9420be3-395c-4e5a-b565-001d9b8cf7f6-kube-api-access-rfgpk\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.095780 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.095715 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.095887 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.095833 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.095887 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.095857 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.096022 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.095885 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.096022 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.095924 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.096149 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.096017 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.096336 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.096312 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.096485 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.096344 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.096608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.096416 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.096718 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.096642 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.098445 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.098420 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.098735 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.098714 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.153420 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.153391 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfgpk\" (UniqueName: \"kubernetes.io/projected/a9420be3-395c-4e5a-b565-001d9b8cf7f6-kube-api-access-rfgpk\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.155025 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.155009 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dff2f\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.159566 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.159545 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:04.392809 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.392735 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:04.601165 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.601134 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f"] Oct 30 19:47:04.606604 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:47:04.606465 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9420be3_395c_4e5a_b565_001d9b8cf7f6.slice/crio-7dabe3bc7e971dadaf76f673c11ebe885606432c815cb4b3c9dfde77c3359d0a WatchSource:0}: Error finding container 7dabe3bc7e971dadaf76f673c11ebe885606432c815cb4b3c9dfde77c3359d0a: Status 404 returned error can't find the container with id 7dabe3bc7e971dadaf76f673c11ebe885606432c815cb4b3c9dfde77c3359d0a Oct 30 19:47:04.624707 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.624651 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:04.629604 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.627729 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:04.629604 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.627829 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:04.629604 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:04.627900 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:05.235323 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:05.235285 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" event={"ID":"a9420be3-395c-4e5a-b565-001d9b8cf7f6","Type":"ContainerStarted","Data":"25e2fe728004d5f187a614801b42224d9d87b9ad12dfda671bad5cc07f2986aa"} Oct 30 19:47:05.236223 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:05.235792 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" event={"ID":"a9420be3-395c-4e5a-b565-001d9b8cf7f6","Type":"ContainerStarted","Data":"7dabe3bc7e971dadaf76f673c11ebe885606432c815cb4b3c9dfde77c3359d0a"} Oct 30 19:47:06.160695 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:06.160655 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:06.162673 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:06.162641 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dff2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:47:06.162823 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:06.162700 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" podUID="a9420be3-395c-4e5a-b565-001d9b8cf7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:47:07.160402 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:07.160367 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dff2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:47:07.160794 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:07.160417 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" podUID="a9420be3-395c-4e5a-b565-001d9b8cf7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:47:08.109800 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:08.107718 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:08.160261 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:08.160227 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dff2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:47:08.160435 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:08.160282 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" podUID="a9420be3-395c-4e5a-b565-001d9b8cf7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:47:09.160489 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:09.160454 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dff2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:47:09.160874 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:09.160507 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" podUID="a9420be3-395c-4e5a-b565-001d9b8cf7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:47:10.160873 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:10.160836 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dff2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:47:10.161291 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:10.160889 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" podUID="a9420be3-395c-4e5a-b565-001d9b8cf7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:47:11.160453 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:11.160413 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dff2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:47:11.160653 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:11.160482 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" podUID="a9420be3-395c-4e5a-b565-001d9b8cf7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:47:12.160658 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:12.160620 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dff2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:47:12.161142 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:12.160690 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" podUID="a9420be3-395c-4e5a-b565-001d9b8cf7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:47:12.982336 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:12.982288 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:13.180227 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:13.180197 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:13.180929 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:13.180907 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:13.181634 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:13.181612 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:47:13.351842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:13.351795 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" podStartSLOduration=10.351783252 podStartE2EDuration="10.351783252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:05.367737767 +0000 UTC m=+1872.302525297" watchObservedRunningTime="2026-10-30 19:47:13.351783252 +0000 UTC m=+1880.286570770" Oct 30 19:47:16.188176 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:16.188137 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:16.774966 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:16.774926 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:17.399655 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:17.399607 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:17.843145 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:17.842799 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:18.272954 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:18.272847 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:18.506907 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:18.506871 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:18.782109 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:18.782079 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv"] Oct 30 19:47:18.786818 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:18.786796 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:18.893836 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:18.893798 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv"] Oct 30 19:47:18.944233 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:18.944201 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-dd9b8d79-openshift-default-dockercfg-nk77p\"" Oct 30 19:47:18.975334 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:18.975299 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:18.975566 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:18.975546 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff913782-d7e9-4110-a499-623fda834acd-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:18.975715 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:18.975703 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff913782-d7e9-4110-a499-623fda834acd-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:18.975835 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:18.975822 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ddkx\" (UniqueName: \"kubernetes.io/projected/ff913782-d7e9-4110-a499-623fda834acd-kube-api-access-8ddkx\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:18.975950 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:18.975938 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:18.976123 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:18.976107 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:18.976244 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:18.976232 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff913782-d7e9-4110-a499-623fda834acd-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:18.976449 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:18.976435 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:18.976575 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:18.976559 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.001436 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.001389 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:19.078845 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.078812 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff913782-d7e9-4110-a499-623fda834acd-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.079104 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.078877 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ddkx\" (UniqueName: \"kubernetes.io/projected/ff913782-d7e9-4110-a499-623fda834acd-kube-api-access-8ddkx\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.079104 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.078927 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.079104 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.079064 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.079298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.079115 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff913782-d7e9-4110-a499-623fda834acd-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.079298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.079226 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.079298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.079260 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.079445 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.079361 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.079445 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.079396 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff913782-d7e9-4110-a499-623fda834acd-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.080020 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.079947 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.080123 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.080059 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff913782-d7e9-4110-a499-623fda834acd-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.080261 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.080242 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.080333 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.080266 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.080477 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.080459 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.082079 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.082054 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.082393 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.082339 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff913782-d7e9-4110-a499-623fda834acd-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.142921 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.142888 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff913782-d7e9-4110-a499-623fda834acd-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.143283 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.143252 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ddkx\" (UniqueName: \"kubernetes.io/projected/ff913782-d7e9-4110-a499-623fda834acd-kube-api-access-8ddkx\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.264432 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.264397 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:19.404699 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.404602 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:19.863165 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.863132 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv"] Oct 30 19:47:19.903432 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:47:19.903405 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff913782_d7e9_4110_a499_623fda834acd.slice/crio-bb486eca0fa5940cb3680d169bd8b23d0f220fa522b42a538b50b8fde7217dc6 WatchSource:0}: Error finding container bb486eca0fa5940cb3680d169bd8b23d0f220fa522b42a538b50b8fde7217dc6: Status 404 returned error can't find the container with id bb486eca0fa5940cb3680d169bd8b23d0f220fa522b42a538b50b8fde7217dc6 Oct 30 19:47:19.905862 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.905829 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:19.905935 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.905904 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:19.905970 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:19.905933 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:20.346196 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:20.346159 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" event={"ID":"ff913782-d7e9-4110-a499-623fda834acd","Type":"ContainerStarted","Data":"b838f891e6cb72d77fdf752fe5f534f2e59d77bb0f6a1e138db7b8e64c0a96e6"} Oct 30 19:47:20.346371 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:20.346222 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" event={"ID":"ff913782-d7e9-4110-a499-623fda834acd","Type":"ContainerStarted","Data":"bb486eca0fa5940cb3680d169bd8b23d0f220fa522b42a538b50b8fde7217dc6"} Oct 30 19:47:20.405001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:20.404959 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:20.410973 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:20.410944 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:20.515734 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:20.515676 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" podStartSLOduration=2.515657267 podStartE2EDuration="2.515657267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:20.510059332 +0000 UTC m=+1887.444846856" watchObservedRunningTime="2026-10-30 19:47:20.515657267 +0000 UTC m=+1887.450444792" Oct 30 19:47:21.353130 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:21.353094 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:21.354259 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:21.354234 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:47:22.227407 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.227377 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:22.255143 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.255111 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n"] Oct 30 19:47:22.260144 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.260122 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:22.312445 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.312419 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8a3c7851-serving-cert\"" Oct 30 19:47:22.336677 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.336649 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d340065c-df73-4f25-bcf9-2b984bdfd710-openshift-service-ca\") pod \"maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:22.336820 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.336806 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-metrics-certs\") pod \"maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:22.336940 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.336927 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-maas-api-tls\") pod \"maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:22.337006 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.336974 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-696x4\" (UniqueName: \"kubernetes.io/projected/d340065c-df73-4f25-bcf9-2b984bdfd710-kube-api-access-696x4\") pod \"maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:22.366272 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.366247 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n"] Oct 30 19:47:22.438433 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.438404 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-metrics-certs\") pod \"maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:22.438617 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.438596 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-maas-api-tls\") pod \"maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:22.438688 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.438638 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696x4\" (UniqueName: \"kubernetes.io/projected/d340065c-df73-4f25-bcf9-2b984bdfd710-kube-api-access-696x4\") pod \"maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:22.438764 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.438707 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d340065c-df73-4f25-bcf9-2b984bdfd710-openshift-service-ca\") pod \"maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:22.439093 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:47:22.439065 4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-serving-cert: secret "maas-api-e2e-derive-8a3c7851-serving-cert" not found Oct 30 19:47:22.439223 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:47:22.439136 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-maas-api-tls podName:d340065c-df73-4f25-bcf9-2b984bdfd710 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:22.939120911 +0000 UTC m=+1889.873908414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-maas-api-tls") pod "maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" (UID: "d340065c-df73-4f25-bcf9-2b984bdfd710") : secret "maas-api-e2e-derive-8a3c7851-serving-cert" not found Oct 30 19:47:22.439449 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.439429 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d340065c-df73-4f25-bcf9-2b984bdfd710-openshift-service-ca\") pod \"maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:22.441294 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.441276 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-metrics-certs\") pod \"maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:22.499011 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.498936 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-696x4\" (UniqueName: \"kubernetes.io/projected/d340065c-df73-4f25-bcf9-2b984bdfd710-kube-api-access-696x4\") pod \"maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:22.538688 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.538656 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:22.765635 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.765543 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:22.946696 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.946665 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-maas-api-tls\") pod \"maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:22.949239 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:22.949213 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-maas-api-tls\") pod \"maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:23.175472 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:23.175435 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:23.272838 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:23.272807 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:23.406715 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:47:23.406675 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:47:23.589836 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:23.589790 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:23.648956 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:23.648928 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n"] Oct 30 19:47:23.651570 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:47:23.651536 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd340065c_df73_4f25_bcf9_2b984bdfd710.slice/crio-a14d7932bbc04a3d22d7fff8f768584bd6b6f1d4e8fa15c07f8cd426a046a8a4 WatchSource:0}: Error finding container a14d7932bbc04a3d22d7fff8f768584bd6b6f1d4e8fa15c07f8cd426a046a8a4: Status 404 returned error can't find the container with id a14d7932bbc04a3d22d7fff8f768584bd6b6f1d4e8fa15c07f8cd426a046a8a4 Oct 30 19:47:24.164363 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:24.164333 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:24.392634 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:24.392591 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" event={"ID":"d340065c-df73-4f25-bcf9-2b984bdfd710","Type":"ContainerStarted","Data":"a14d7932bbc04a3d22d7fff8f768584bd6b6f1d4e8fa15c07f8cd426a046a8a4"} Oct 30 19:47:24.394028 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:24.393972 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:25.400749 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:25.400711 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" event={"ID":"d340065c-df73-4f25-bcf9-2b984bdfd710","Type":"ContainerStarted","Data":"cf7d43dbdd9b11749b1672522f0676022eadfb7ecdeb64ea82b92e72d1cd9bba"} Oct 30 19:47:25.401209 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:25.400789 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:25.515716 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:25.515660 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" podStartSLOduration=2.724194329 podStartE2EDuration="3.515645652s" totalImagesPullingTime="791.451323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:25.514167799 +0000 UTC m=+1892.448955324" watchObservedRunningTime="2026-10-30 19:47:25.515645652 +0000 UTC m=+1892.450433175" Oct 30 19:47:27.959876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:27.959842 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:28.805838 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:28.805802 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:29.086615 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:29.086548 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:29.362055 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:29.361938 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:30.059177 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:30.059146 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:30.295134 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:30.295098 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:31.412551 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:31.412525 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:47:31.478315 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:31.478284 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:31.869686 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:31.869652 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:33.011148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.011116 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz"] Oct 30 19:47:33.015663 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.015646 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:33.064797 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.064768 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-metrics-certs\") pod \"maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:33.064919 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.064898 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-maas-api-tls\") pod \"maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:33.064961 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.064935 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f4fp\" (UniqueName: \"kubernetes.io/projected/95740794-649d-4a88-9f4b-ada9e9fe7d2e-kube-api-access-7f4fp\") pod \"maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:33.065039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.065012 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95740794-649d-4a88-9f4b-ada9e9fe7d2e-openshift-service-ca\") pod \"maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:33.068569 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.068547 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-df789477-serving-cert\"" Oct 30 19:47:33.074212 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.074189 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz"] Oct 30 19:47:33.166416 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.166382 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-maas-api-tls\") pod \"maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:33.166602 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.166444 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f4fp\" (UniqueName: \"kubernetes.io/projected/95740794-649d-4a88-9f4b-ada9e9fe7d2e-kube-api-access-7f4fp\") pod \"maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:33.166602 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:47:33.166476 4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-df789477-serving-cert: secret "maas-api-e2e-ait-df789477-serving-cert" not found Oct 30 19:47:33.166602 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:47:33.166536 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-maas-api-tls podName:95740794-649d-4a88-9f4b-ada9e9fe7d2e nodeName:}" failed. No retries permitted until 2026-10-30 19:47:33.666516776 +0000 UTC m=+1900.601304287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-maas-api-tls") pod "maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" (UID: "95740794-649d-4a88-9f4b-ada9e9fe7d2e") : secret "maas-api-e2e-ait-df789477-serving-cert" not found Oct 30 19:47:33.166602 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.166559 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95740794-649d-4a88-9f4b-ada9e9fe7d2e-openshift-service-ca\") pod \"maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:33.166915 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.166836 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-metrics-certs\") pod \"maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:33.167760 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.167733 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95740794-649d-4a88-9f4b-ada9e9fe7d2e-openshift-service-ca\") pod \"maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:33.169639 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.169615 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-metrics-certs\") pod \"maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:33.224672 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.224651 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f4fp\" (UniqueName: \"kubernetes.io/projected/95740794-649d-4a88-9f4b-ada9e9fe7d2e-kube-api-access-7f4fp\") pod \"maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:33.293383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.293274 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:33.556719 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.556288 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:33.677688 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.677656 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-maas-api-tls\") pod \"maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:33.680064 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.680035 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-maas-api-tls\") pod \"maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:33.799672 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.799630 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:33.929105 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:33.929066 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:34.309640 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:34.309613 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz"] Oct 30 19:47:34.310442 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:47:34.310417 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95740794_649d_4a88_9f4b_ada9e9fe7d2e.slice/crio-48d5b6c7868874681ea14bbe8a6b6e078b5b5cd72a4a23217c0c3a2c891db879 WatchSource:0}: Error finding container 48d5b6c7868874681ea14bbe8a6b6e078b5b5cd72a4a23217c0c3a2c891db879: Status 404 returned error can't find the container with id 48d5b6c7868874681ea14bbe8a6b6e078b5b5cd72a4a23217c0c3a2c891db879 Oct 30 19:47:34.462308 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:34.462274 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" event={"ID":"95740794-649d-4a88-9f4b-ada9e9fe7d2e","Type":"ContainerStarted","Data":"48d5b6c7868874681ea14bbe8a6b6e078b5b5cd72a4a23217c0c3a2c891db879"} Oct 30 19:47:35.471498 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:35.471386 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" event={"ID":"95740794-649d-4a88-9f4b-ada9e9fe7d2e","Type":"ContainerStarted","Data":"d036fecf0aaf0850d02f6e68b1cb6d156c68a20f540db6d8d791b39ed03c527a"} Oct 30 19:47:35.471498 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:35.471468 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:35.587301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:35.587251 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" podStartSLOduration=2.809847662 podStartE2EDuration="3.5872378s" totalImagesPullingTime="777.390138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:35.585592186 +0000 UTC m=+1902.520379708" watchObservedRunningTime="2026-10-30 19:47:35.5872378 +0000 UTC m=+1902.522025323" Oct 30 19:47:38.350590 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:38.350555 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:39.214544 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:39.214495 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:39.445482 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:39.445445 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:41.516683 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:41.516656 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:47:43.050865 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:43.050833 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:43.274781 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:43.274732 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:43.805333 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:43.805291 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:44.230300 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:44.230261 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:44.459044 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:44.459009 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:47.809837 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:47.809805 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:50.827023 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:50.826959 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:51.007114 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:51.007062 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gtk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:47:51.007413 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:51.007371 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:47:52.584781 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:52.584747 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:52.904401 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:52.904304 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:53.366997 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:47:53.366949 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:47:57.304917 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:57.304879 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:58.085672 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:58.085637 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:58.323339 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:47:58.323307 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:01.953695 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:01.953656 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:02.182505 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.182465 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:02.333370 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.333333 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb"] Oct 30 19:48:02.353449 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.353418 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:02.407075 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.407047 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-c75ed4-serving-cert\"" Oct 30 19:48:02.441070 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.441031 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:02.441360 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.441089 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17419a84-960d-4236-90c3-debde62918d9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:02.441360 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.441202 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlvpc\" (UniqueName: \"kubernetes.io/projected/17419a84-960d-4236-90c3-debde62918d9-kube-api-access-wlvpc\") pod \"maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:02.470050 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.448270 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:02.470483 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.468149 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb"] Oct 30 19:48:02.573654 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.573608 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:02.574236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.573667 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17419a84-960d-4236-90c3-debde62918d9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:02.574236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.573760 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvpc\" (UniqueName: \"kubernetes.io/projected/17419a84-960d-4236-90c3-debde62918d9-kube-api-access-wlvpc\") pod \"maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:02.574236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.573855 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:02.574236 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:48:02.574048 4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-serving-cert: secret "maas-api-e2e-shared-b-w3-c75ed4-serving-cert" not found Oct 30 19:48:02.574236 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:48:02.574125 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-maas-api-tls podName:17419a84-960d-4236-90c3-debde62918d9 nodeName:}" failed. No retries permitted until 2026-10-30 19:48:03.074103203 +0000 UTC m=+1930.008890728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-maas-api-tls") pod "maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" (UID: "17419a84-960d-4236-90c3-debde62918d9") : secret "maas-api-e2e-shared-b-w3-c75ed4-serving-cert" not found Oct 30 19:48:02.575916 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.575888 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17419a84-960d-4236-90c3-debde62918d9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:02.579020 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.578968 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:02.653047 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.652359 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvpc\" (UniqueName: \"kubernetes.io/projected/17419a84-960d-4236-90c3-debde62918d9-kube-api-access-wlvpc\") pod \"maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:02.673019 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.672064 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:02.909341 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:02.909255 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:03.081693 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:03.081648 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:03.084301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:03.084268 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:03.269491 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:03.269404 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:03.650568 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:03.650543 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb"] Oct 30 19:48:03.651685 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:48:03.651662 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17419a84_960d_4236_90c3_debde62918d9.slice/crio-2d08766e76c06fda19b5b4defae2fa3e2aa17206a25bfecd615363b2d3be871c WatchSource:0}: Error finding container 2d08766e76c06fda19b5b4defae2fa3e2aa17206a25bfecd615363b2d3be871c: Status 404 returned error can't find the container with id 2d08766e76c06fda19b5b4defae2fa3e2aa17206a25bfecd615363b2d3be871c Oct 30 19:48:03.785954 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:03.785925 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" event={"ID":"17419a84-960d-4236-90c3-debde62918d9","Type":"ContainerStarted","Data":"2d08766e76c06fda19b5b4defae2fa3e2aa17206a25bfecd615363b2d3be871c"} Oct 30 19:48:04.794871 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:04.794778 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" event={"ID":"17419a84-960d-4236-90c3-debde62918d9","Type":"ContainerStarted","Data":"68a227d87457f866d9f4fc555d6b64f9f4213ad09ef0bc9795e63cc9ccabaac5"} Oct 30 19:48:04.794871 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:04.794842 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:04.908608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:04.908561 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" podStartSLOduration=2.103390476 podStartE2EDuration="2.908546865s" totalImagesPullingTime="805.156389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:04.908115076 +0000 UTC m=+1931.842902612" watchObservedRunningTime="2026-10-30 19:48:04.908546865 +0000 UTC m=+1931.843334389" Oct 30 19:48:07.567433 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:07.567398 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:10.808751 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:10.808724 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 19:48:12.165586 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:12.165547 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:16.516432 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:16.516399 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:16.776709 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:16.776608 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:21.396283 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:21.396248 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:23.367477 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:48:23.367440 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:48:26.070225 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:26.070187 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:26.298699 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:26.298659 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:26.545119 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:26.544952 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:28.505103 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:28.505047 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pmt9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:48:28.505754 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:28.505716 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" podUID="f7f198b8-0ce4-4ac4-9339-944e7f829bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:28.619031 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:28.601159 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:28.661215 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:28.661148 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:28.801410 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:28.801290 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:28.928172 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:28.928118 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" podUID="eafb6a50-b7db-47d6-b742-86b92bfef67d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:28.974723 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:28.973063 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" podUID="7f1295b7-8682-4eed-b904-83b8b6537e8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:31.516487 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:31.516449 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:35.101132 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:35.101086 4433 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:48:35.101713 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:35.101157 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" podUID="efa7df32-302f-44e6-aef2-c4aff86c2b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:36.428046 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:36.427960 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:36.681514 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:36.681418 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:37.119412 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:37.119381 4433 generic.go:356] "Generic (PLEG): container finished" podID="efa7df32-302f-44e6-aef2-c4aff86c2b78" containerID="2841278e2d0b1dc47a4ca7dcea4fb66c1f7248b8418702608e13c991bd27ac41" exitCode=0 Oct 30 19:48:37.119543 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:37.119448 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" event={"ID":"efa7df32-302f-44e6-aef2-c4aff86c2b78","Type":"ContainerDied","Data":"2841278e2d0b1dc47a4ca7dcea4fb66c1f7248b8418702608e13c991bd27ac41"} Oct 30 19:48:37.119833 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:37.119819 4433 scope.go:122] "RemoveContainer" containerID="2841278e2d0b1dc47a4ca7dcea4fb66c1f7248b8418702608e13c991bd27ac41" Oct 30 19:48:37.121463 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:37.121436 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:48:37.121546 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:37.121506 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:48:37.121586 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:37.121543 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:48:38.136866 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:38.136824 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" event={"ID":"efa7df32-302f-44e6-aef2-c4aff86c2b78","Type":"ContainerStarted","Data":"bf43165051170dc959034f442f33f28976f4de0c826b1d0e5e998c6960a6ecbc"} Oct 30 19:48:38.432023 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:38.431894 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:38.619707 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:38.619667 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:48:38.619890 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:38.619838 4433 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:48:38.619937 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:38.619897 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" podUID="efa7df32-302f-44e6-aef2-c4aff86c2b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 19:48:38.666956 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:38.666926 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:39.624009 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:39.623348 4433 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:48:39.624009 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:39.623419 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" podUID="efa7df32-302f-44e6-aef2-c4aff86c2b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 19:48:40.619698 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:40.619654 4433 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:48:40.619871 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:40.619715 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" podUID="efa7df32-302f-44e6-aef2-c4aff86c2b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 19:48:41.380561 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:41.380492 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:41.619903 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:41.619868 4433 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:48:41.620125 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:41.619932 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" podUID="efa7df32-302f-44e6-aef2-c4aff86c2b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 19:48:42.620135 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:42.620098 4433 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:48:42.620509 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:42.620149 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" podUID="efa7df32-302f-44e6-aef2-c4aff86c2b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 19:48:43.624111 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:43.624078 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:48:43.624492 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:43.624458 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:48:43.625329 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:43.625310 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 19:48:46.008111 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:46.008080 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:47.257159 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:47.257116 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9"] Oct 30 19:48:47.410830 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:47.410801 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg"] Oct 30 19:48:47.767881 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:47.767832 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:48.005264 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:48.005231 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:50.882109 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:50.882077 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:52.258374 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:52.258305 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" podUID="1b11cfae-71ff-4a16-96e3-e1fe6848eb46" containerName="payload-processing" containerID="cri-o://1869633ca601aa746c3e34e99de4d1cfb3600e09ad0bd2149d1cb951531d9cdf" gracePeriod=25 Oct 30 19:48:52.411282 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:52.411241 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" podUID="104e9da9-96a0-426b-9363-d5885dd2fe63" containerName="payload-pre-processing" containerID="cri-o://f5ccd83f6b59ef9d589e0a48a046cc9eb2d910e833c1e998e6d47aaf417173f1" gracePeriod=25 Oct 30 19:48:52.638173 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:52.637771 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" Oct 30 19:48:52.749868 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:52.749843 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" Oct 30 19:48:52.775716 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:52.775686 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b11cfae-71ff-4a16-96e3-e1fe6848eb46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b11cfae-71ff-4a16-96e3-e1fe6848eb46-plugins-config-volume\") pod \"1b11cfae-71ff-4a16-96e3-e1fe6848eb46\" (UID: \"1b11cfae-71ff-4a16-96e3-e1fe6848eb46\") " Oct 30 19:48:52.775863 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:52.775826 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b11cfae-71ff-4a16-96e3-e1fe6848eb46-kube-api-access-bljtj\" (UniqueName: \"kubernetes.io/projected/1b11cfae-71ff-4a16-96e3-e1fe6848eb46-kube-api-access-bljtj\") pod \"1b11cfae-71ff-4a16-96e3-e1fe6848eb46\" (UID: \"1b11cfae-71ff-4a16-96e3-e1fe6848eb46\") " Oct 30 19:48:52.776134 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:52.776109 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b11cfae-71ff-4a16-96e3-e1fe6848eb46-plugins-config-volume" pod "1b11cfae-71ff-4a16-96e3-e1fe6848eb46" (UID: "1b11cfae-71ff-4a16-96e3-e1fe6848eb46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:52.776520 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:52.776498 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b11cfae-71ff-4a16-96e3-e1fe6848eb46-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:48:52.778015 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:52.777950 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b11cfae-71ff-4a16-96e3-e1fe6848eb46-kube-api-access-bljtj" pod "1b11cfae-71ff-4a16-96e3-e1fe6848eb46" (UID: "1b11cfae-71ff-4a16-96e3-e1fe6848eb46"). InnerVolumeSpecName "kube-api-access-bljtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:52.877619 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:52.877586 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/104e9da9-96a0-426b-9363-d5885dd2fe63-kube-api-access-qrpwb\" (UniqueName: \"kubernetes.io/projected/104e9da9-96a0-426b-9363-d5885dd2fe63-kube-api-access-qrpwb\") pod \"104e9da9-96a0-426b-9363-d5885dd2fe63\" (UID: \"104e9da9-96a0-426b-9363-d5885dd2fe63\") " Oct 30 19:48:52.877795 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:52.877750 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/104e9da9-96a0-426b-9363-d5885dd2fe63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104e9da9-96a0-426b-9363-d5885dd2fe63-plugins-config-volume\") pod \"104e9da9-96a0-426b-9363-d5885dd2fe63\" (UID: \"104e9da9-96a0-426b-9363-d5885dd2fe63\") " Oct 30 19:48:52.878318 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:52.878286 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104e9da9-96a0-426b-9363-d5885dd2fe63-plugins-config-volume" pod "104e9da9-96a0-426b-9363-d5885dd2fe63" (UID: "104e9da9-96a0-426b-9363-d5885dd2fe63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:52.878443 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:52.878429 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104e9da9-96a0-426b-9363-d5885dd2fe63-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:48:52.878510 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:52.878451 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bljtj\" (UniqueName: \"kubernetes.io/projected/1b11cfae-71ff-4a16-96e3-e1fe6848eb46-kube-api-access-bljtj\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:48:52.880512 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:52.880483 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104e9da9-96a0-426b-9363-d5885dd2fe63-kube-api-access-qrpwb" pod "104e9da9-96a0-426b-9363-d5885dd2fe63" (UID: "104e9da9-96a0-426b-9363-d5885dd2fe63"). InnerVolumeSpecName "kube-api-access-qrpwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:52.980238 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:52.980202 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrpwb\" (UniqueName: \"kubernetes.io/projected/104e9da9-96a0-426b-9363-d5885dd2fe63-kube-api-access-qrpwb\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:48:53.277228 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.277203 4433 generic.go:356] "Generic (PLEG): container finished" podID="1b11cfae-71ff-4a16-96e3-e1fe6848eb46" containerID="1869633ca601aa746c3e34e99de4d1cfb3600e09ad0bd2149d1cb951531d9cdf" exitCode=0 Oct 30 19:48:53.277571 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.277257 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" Oct 30 19:48:53.277571 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.277290 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" event={"ID":"1b11cfae-71ff-4a16-96e3-e1fe6848eb46","Type":"ContainerDied","Data":"1869633ca601aa746c3e34e99de4d1cfb3600e09ad0bd2149d1cb951531d9cdf"} Oct 30 19:48:53.277571 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.277350 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9" event={"ID":"1b11cfae-71ff-4a16-96e3-e1fe6848eb46","Type":"ContainerDied","Data":"01f89ac4e15d688b2062303214f8701666a9a8a85e8c8a6ba5b1c89e09b0d28b"} Oct 30 19:48:53.277571 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.277381 4433 scope.go:122] "RemoveContainer" containerID="1869633ca601aa746c3e34e99de4d1cfb3600e09ad0bd2149d1cb951531d9cdf" Oct 30 19:48:53.279177 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.279152 4433 generic.go:356] "Generic (PLEG): container finished" podID="104e9da9-96a0-426b-9363-d5885dd2fe63" containerID="f5ccd83f6b59ef9d589e0a48a046cc9eb2d910e833c1e998e6d47aaf417173f1" exitCode=0 Oct 30 19:48:53.279295 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.279199 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" Oct 30 19:48:53.279295 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.279274 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" event={"ID":"104e9da9-96a0-426b-9363-d5885dd2fe63","Type":"ContainerDied","Data":"f5ccd83f6b59ef9d589e0a48a046cc9eb2d910e833c1e998e6d47aaf417173f1"} Oct 30 19:48:53.279417 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.279322 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg" event={"ID":"104e9da9-96a0-426b-9363-d5885dd2fe63","Type":"ContainerDied","Data":"377a9675a9944c6ceaa7700b4decb982a4e89f22d2ac5e343b6b9b36170e724c"} Oct 30 19:48:53.316370 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.316335 4433 scope.go:122] "RemoveContainer" containerID="1869633ca601aa746c3e34e99de4d1cfb3600e09ad0bd2149d1cb951531d9cdf" Oct 30 19:48:53.316904 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:48:53.316879 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1869633ca601aa746c3e34e99de4d1cfb3600e09ad0bd2149d1cb951531d9cdf\": container with ID starting with 1869633ca601aa746c3e34e99de4d1cfb3600e09ad0bd2149d1cb951531d9cdf not found: ID does not exist" containerID="1869633ca601aa746c3e34e99de4d1cfb3600e09ad0bd2149d1cb951531d9cdf" Oct 30 19:48:53.317039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.316910 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1869633ca601aa746c3e34e99de4d1cfb3600e09ad0bd2149d1cb951531d9cdf"} err="failed to get container status \"1869633ca601aa746c3e34e99de4d1cfb3600e09ad0bd2149d1cb951531d9cdf\": rpc error: code = NotFound desc = could not find container \"1869633ca601aa746c3e34e99de4d1cfb3600e09ad0bd2149d1cb951531d9cdf\": container with ID starting with 1869633ca601aa746c3e34e99de4d1cfb3600e09ad0bd2149d1cb951531d9cdf not found: ID does not exist" Oct 30 19:48:53.317039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.316927 4433 scope.go:122] "RemoveContainer" containerID="f5ccd83f6b59ef9d589e0a48a046cc9eb2d910e833c1e998e6d47aaf417173f1" Oct 30 19:48:53.326082 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.326065 4433 scope.go:122] "RemoveContainer" containerID="f5ccd83f6b59ef9d589e0a48a046cc9eb2d910e833c1e998e6d47aaf417173f1" Oct 30 19:48:53.326306 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:48:53.326288 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ccd83f6b59ef9d589e0a48a046cc9eb2d910e833c1e998e6d47aaf417173f1\": container with ID starting with f5ccd83f6b59ef9d589e0a48a046cc9eb2d910e833c1e998e6d47aaf417173f1 not found: ID does not exist" containerID="f5ccd83f6b59ef9d589e0a48a046cc9eb2d910e833c1e998e6d47aaf417173f1" Oct 30 19:48:53.326368 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.326314 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ccd83f6b59ef9d589e0a48a046cc9eb2d910e833c1e998e6d47aaf417173f1"} err="failed to get container status \"f5ccd83f6b59ef9d589e0a48a046cc9eb2d910e833c1e998e6d47aaf417173f1\": rpc error: code = NotFound desc = could not find container \"f5ccd83f6b59ef9d589e0a48a046cc9eb2d910e833c1e998e6d47aaf417173f1\": container with ID starting with f5ccd83f6b59ef9d589e0a48a046cc9eb2d910e833c1e998e6d47aaf417173f1 not found: ID does not exist" Oct 30 19:48:53.367399 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:48:53.367366 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:48:53.441673 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.441639 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg"] Oct 30 19:48:53.513729 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.513652 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8a3c7851-7784555567-jp9mg"] Oct 30 19:48:53.567735 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.567704 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9"] Oct 30 19:48:53.571214 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:53.571192 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8a3c7851-78ff8c5956-xtcv9"] Oct 30 19:48:55.216020 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:55.215973 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="104e9da9-96a0-426b-9363-d5885dd2fe63" path="/var/lib/kubelet/pods/104e9da9-96a0-426b-9363-d5885dd2fe63/volumes" Oct 30 19:48:55.216927 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:55.216902 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b11cfae-71ff-4a16-96e3-e1fe6848eb46" path="/var/lib/kubelet/pods/1b11cfae-71ff-4a16-96e3-e1fe6848eb46/volumes" Oct 30 19:48:55.942457 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:55.941546 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:59.340656 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:59.340625 4433 generic.go:356] "Generic (PLEG): container finished" podID="ff913782-d7e9-4110-a499-623fda834acd" containerID="b838f891e6cb72d77fdf752fe5f534f2e59d77bb0f6a1e138db7b8e64c0a96e6" exitCode=0 Oct 30 19:48:59.341090 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:59.340688 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" event={"ID":"ff913782-d7e9-4110-a499-623fda834acd","Type":"ContainerDied","Data":"b838f891e6cb72d77fdf752fe5f534f2e59d77bb0f6a1e138db7b8e64c0a96e6"} Oct 30 19:48:59.341090 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:59.341032 4433 scope.go:122] "RemoveContainer" containerID="b838f891e6cb72d77fdf752fe5f534f2e59d77bb0f6a1e138db7b8e64c0a96e6" Oct 30 19:48:59.342408 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:59.342382 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:48:59.342470 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:59.342450 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:48:59.342506 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:48:59.342482 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:00.351169 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:00.351135 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" event={"ID":"ff913782-d7e9-4110-a499-623fda834acd","Type":"ContainerStarted","Data":"14c0c02aaacf0e9d4108c0990c8304d2bee4aacb801f64b9b85abc28bd9950eb"} Oct 30 19:49:00.409005 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:00.406361 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:49:00.409005 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:00.407260 4433 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:49:00.409005 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:00.407332 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" podUID="ff913782-d7e9-4110-a499-623fda834acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 19:49:00.976293 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:00.976258 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:01.409333 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:01.409297 4433 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:49:01.409893 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:01.409850 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" podUID="ff913782-d7e9-4110-a499-623fda834acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 19:49:02.409465 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:02.409413 4433 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:49:02.409909 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:02.409484 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" podUID="ff913782-d7e9-4110-a499-623fda834acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 19:49:03.408001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:03.407253 4433 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:49:03.408001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:03.407309 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" podUID="ff913782-d7e9-4110-a499-623fda834acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 19:49:04.406741 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:04.406704 4433 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:49:04.407309 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:04.407276 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" podUID="ff913782-d7e9-4110-a499-623fda834acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 19:49:05.409699 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:05.409667 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:49:05.410099 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:05.410081 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:49:05.410611 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:05.410586 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 19:49:06.050415 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:06.050374 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:09.724271 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:09.724241 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw"] Oct 30 19:49:09.724719 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:09.724421 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" podUID="07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" containerName="istio-proxy" containerID="cri-o://eb2353eeb6fe16757e609eefeabbdc4fb1de4b5d5be097cdeb83e32cc962d013" gracePeriod=30 Oct 30 19:49:10.727252 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.727219 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz"] Oct 30 19:49:10.727896 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.727880 4433 cpu_manager.go:414] "containerMap: removing container" podUID="1b11cfae-71ff-4a16-96e3-e1fe6848eb46" containerName="payload-processing" Oct 30 19:49:10.727944 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.727897 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b11cfae-71ff-4a16-96e3-e1fe6848eb46" containerName="payload-processing" Oct 30 19:49:10.727944 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.727933 4433 cpu_manager.go:414] "containerMap: removing container" podUID="104e9da9-96a0-426b-9363-d5885dd2fe63" containerName="payload-pre-processing" Oct 30 19:49:10.727944 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.727938 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="104e9da9-96a0-426b-9363-d5885dd2fe63" containerName="payload-pre-processing" Oct 30 19:49:10.728104 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.728058 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="104e9da9-96a0-426b-9363-d5885dd2fe63" containerName="payload-pre-processing" Oct 30 19:49:10.728104 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.728070 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b11cfae-71ff-4a16-96e3-e1fe6848eb46" containerName="payload-processing" Oct 30 19:49:10.731972 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.731955 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:10.834241 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.834209 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5bf13ff3-serving-cert\"" Oct 30 19:49:10.838266 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.838240 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz"] Oct 30 19:49:10.895556 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.895530 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-maas-api-tls\") pod \"maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:10.895820 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.895791 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-metrics-certs\") pod \"maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:10.895911 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.895849 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-openshift-service-ca\") pod \"maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:10.896039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.896022 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h958h\" (UniqueName: \"kubernetes.io/projected/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-kube-api-access-h958h\") pod \"maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:10.997234 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.997159 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-maas-api-tls\") pod \"maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:10.997381 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.997263 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-metrics-certs\") pod \"maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:10.997381 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.997301 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-openshift-service-ca\") pod \"maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:10.997381 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:49:10.997318 4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-serving-cert: secret "maas-api-e2e-mt-5bf13ff3-serving-cert" not found Oct 30 19:49:10.997547 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:49:10.997394 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-maas-api-tls podName:3ac90de6-4478-4ba6-b106-d40c7c6be7b9 nodeName:}" failed. No retries permitted until 2026-10-30 19:49:11.497374342 +0000 UTC m=+1998.432161869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-maas-api-tls") pod "maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" (UID: "3ac90de6-4478-4ba6-b106-d40c7c6be7b9") : secret "maas-api-e2e-mt-5bf13ff3-serving-cert" not found Oct 30 19:49:10.997547 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.997439 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h958h\" (UniqueName: \"kubernetes.io/projected/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-kube-api-access-h958h\") pod \"maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:10.998014 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.997991 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-openshift-service-ca\") pod \"maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:10.999808 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:10.999782 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-metrics-certs\") pod \"maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:11.072247 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:11.072204 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h958h\" (UniqueName: \"kubernetes.io/projected/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-kube-api-access-h958h\") pod \"maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:11.077763 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:11.077729 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:11.367638 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:11.367608 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:11.506752 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:11.506713 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-maas-api-tls\") pod \"maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:11.509126 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:11.509105 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-maas-api-tls\") pod \"maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:11.619369 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:11.619303 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:11.645602 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:11.645579 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:12.029836 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:12.029805 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz"] Oct 30 19:49:12.030238 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:49:12.030208 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac90de6_4478_4ba6_b106_d40c7c6be7b9.slice/crio-08b4cfedf08579bab064d922ca06c84e717622f32fc083156e7a534c860e693b WatchSource:0}: Error finding container 08b4cfedf08579bab064d922ca06c84e717622f32fc083156e7a534c860e693b: Status 404 returned error can't find the container with id 08b4cfedf08579bab064d922ca06c84e717622f32fc083156e7a534c860e693b Oct 30 19:49:12.031606 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:12.031584 4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:49:12.436879 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:12.436841 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" event={"ID":"3ac90de6-4478-4ba6-b106-d40c7c6be7b9","Type":"ContainerStarted","Data":"08b4cfedf08579bab064d922ca06c84e717622f32fc083156e7a534c860e693b"} Oct 30 19:49:13.162432 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:13.162385 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:13.445960 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:13.445923 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" event={"ID":"3ac90de6-4478-4ba6-b106-d40c7c6be7b9","Type":"ContainerStarted","Data":"ac9ee728e6e09bcf8970690e13cecd4456fcca3ebe75afc379b1017b44feb37a"} Oct 30 19:49:13.446168 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:13.446007 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:13.565096 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:13.565046 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" podStartSLOduration=2.647775034 podStartE2EDuration="3.565031119s" totalImagesPullingTime="917.256085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:13.561176037 +0000 UTC m=+2000.495963560" watchObservedRunningTime="2026-10-30 19:49:13.565031119 +0000 UTC m=+2000.499818642" Oct 30 19:49:14.814105 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:14.814066 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:49:14.814472 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:14.814125 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" podUID="07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:49:15.002482 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.002414 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:49:15.056806 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.056780 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-token\" (UniqueName: \"kubernetes.io/projected/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-token\") pod \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " Oct 30 19:49:15.056922 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.056827 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-workload-certs\") pod \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " Oct 30 19:49:15.056922 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.056895 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-credential-socket\") pod \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " Oct 30 19:49:15.057044 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.056931 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-kube-api-access-bwl8v\" (UniqueName: \"kubernetes.io/projected/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-kube-api-access-bwl8v\") pod \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " Oct 30 19:49:15.057044 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.057017 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-data\") pod \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " Oct 30 19:49:15.057044 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.057038 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-podinfo\") pod \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " Oct 30 19:49:15.057158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.057088 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-envoy\") pod \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " Oct 30 19:49:15.057158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.057119 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istiod-ca-cert\") pod \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " Oct 30 19:49:15.057158 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.057107 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-workload-certs" pod "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" (UID: "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:15.057309 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.057170 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-credential-socket" pod "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" (UID: "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:15.057361 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.057312 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-data" pod "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" (UID: "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:15.057361 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.057322 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-workload-socket\") pod \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\" (UID: \"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3\") " Oct 30 19:49:15.057582 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.057560 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-workload-socket" pod "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" (UID: "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:15.057662 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.057600 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istiod-ca-cert" pod "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" (UID: "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:15.058148 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.058122 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:15.058236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.058153 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:15.058236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.058172 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:15.058236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.058188 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:15.058236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.058203 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:15.059238 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.059216 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-token" pod "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" (UID: "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:15.059352 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.059331 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-envoy" pod "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" (UID: "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:15.059417 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.059354 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-kube-api-access-bwl8v" pod "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" (UID: "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3"). InnerVolumeSpecName "kube-api-access-bwl8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:15.059417 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.059363 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-podinfo" pod "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" (UID: "07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:49:15.159695 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.159670 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:15.159695 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.159695 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwl8v\" (UniqueName: \"kubernetes.io/projected/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-kube-api-access-bwl8v\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:15.159808 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.159705 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:15.159808 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.159714 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:15.470097 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.470054 4433 generic.go:356] "Generic (PLEG): container finished" podID="07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" containerID="eb2353eeb6fe16757e609eefeabbdc4fb1de4b5d5be097cdeb83e32cc962d013" exitCode=0 Oct 30 19:49:15.470283 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.470126 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" Oct 30 19:49:15.470283 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.470171 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" event={"ID":"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3","Type":"ContainerDied","Data":"eb2353eeb6fe16757e609eefeabbdc4fb1de4b5d5be097cdeb83e32cc962d013"} Oct 30 19:49:15.470283 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.470253 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw" event={"ID":"07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3","Type":"ContainerDied","Data":"83b0db5054b353aace5c7e7ffe8e374eb506428de1b3b2750172b5113156b9e3"} Oct 30 19:49:15.470431 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.470284 4433 scope.go:122] "RemoveContainer" containerID="eb2353eeb6fe16757e609eefeabbdc4fb1de4b5d5be097cdeb83e32cc962d013" Oct 30 19:49:15.484972 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.484938 4433 scope.go:122] "RemoveContainer" containerID="eb2353eeb6fe16757e609eefeabbdc4fb1de4b5d5be097cdeb83e32cc962d013" Oct 30 19:49:15.485425 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:49:15.485401 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb2353eeb6fe16757e609eefeabbdc4fb1de4b5d5be097cdeb83e32cc962d013\": container with ID starting with eb2353eeb6fe16757e609eefeabbdc4fb1de4b5d5be097cdeb83e32cc962d013 not found: ID does not exist" containerID="eb2353eeb6fe16757e609eefeabbdc4fb1de4b5d5be097cdeb83e32cc962d013" Oct 30 19:49:15.485513 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.485434 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb2353eeb6fe16757e609eefeabbdc4fb1de4b5d5be097cdeb83e32cc962d013"} err="failed to get container status \"eb2353eeb6fe16757e609eefeabbdc4fb1de4b5d5be097cdeb83e32cc962d013\": rpc error: code = NotFound desc = could not find container \"eb2353eeb6fe16757e609eefeabbdc4fb1de4b5d5be097cdeb83e32cc962d013\": container with ID starting with eb2353eeb6fe16757e609eefeabbdc4fb1de4b5d5be097cdeb83e32cc962d013 not found: ID does not exist" Oct 30 19:49:15.643934 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.643900 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw"] Oct 30 19:49:15.647710 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:15.647684 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-7mzkw"] Oct 30 19:49:16.450424 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:16.450361 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:17.214975 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:17.214946 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" path="/var/lib/kubelet/pods/07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3/volumes" Oct 30 19:49:19.467323 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:19.467293 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:49:21.234216 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:21.234179 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:22.354118 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:22.354077 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:23.366744 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:49:23.366709 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:49:25.622301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:25.622255 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:26.058422 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:26.058318 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:28.901266 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:28.901225 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:31.045130 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:31.045088 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:32.092617 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:32.092563 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:35.753645 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:35.753552 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:38.316964 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:38.316928 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7"] Oct 30 19:49:38.428731 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:38.428700 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp"] Oct 30 19:49:38.782716 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:38.782584 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:39.039105 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:39.039023 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:40.177876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:40.177528 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:40.482498 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:40.482403 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:43.318232 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.318191 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" podUID="31b8ebe9-1b26-4870-b9ad-6535916b56f7" containerName="payload-processing" containerID="cri-o://9b0baeede1efc97f2c7c3af1e1183dee11ccc7faa1bd4e642ab895ff0f4b5a14" gracePeriod=25 Oct 30 19:49:43.429260 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.429217 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" podUID="009b4710-bd3c-4273-bbee-84ebfbe13896" containerName="payload-pre-processing" containerID="cri-o://0a22cfe8e02833ea8b2cc4c52da3b725fd489bbb84a08dd75dec4469cdb193e0" gracePeriod=25 Oct 30 19:49:43.640589 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.640557 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" Oct 30 19:49:43.679265 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.679229 4433 generic.go:356] "Generic (PLEG): container finished" podID="009b4710-bd3c-4273-bbee-84ebfbe13896" containerID="0a22cfe8e02833ea8b2cc4c52da3b725fd489bbb84a08dd75dec4469cdb193e0" exitCode=0 Oct 30 19:49:43.679431 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.679299 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" event={"ID":"009b4710-bd3c-4273-bbee-84ebfbe13896","Type":"ContainerDied","Data":"0a22cfe8e02833ea8b2cc4c52da3b725fd489bbb84a08dd75dec4469cdb193e0"} Oct 30 19:49:43.681111 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.681087 4433 generic.go:356] "Generic (PLEG): container finished" podID="31b8ebe9-1b26-4870-b9ad-6535916b56f7" containerID="9b0baeede1efc97f2c7c3af1e1183dee11ccc7faa1bd4e642ab895ff0f4b5a14" exitCode=0 Oct 30 19:49:43.681236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.681196 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" Oct 30 19:49:43.681478 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.681119 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" event={"ID":"31b8ebe9-1b26-4870-b9ad-6535916b56f7","Type":"ContainerDied","Data":"9b0baeede1efc97f2c7c3af1e1183dee11ccc7faa1bd4e642ab895ff0f4b5a14"} Oct 30 19:49:43.681552 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.681523 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7" event={"ID":"31b8ebe9-1b26-4870-b9ad-6535916b56f7","Type":"ContainerDied","Data":"bfbe2a827727461c396ff1efd27e6a53710903231220ffea5353cbc458134be5"} Oct 30 19:49:43.681615 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.681581 4433 scope.go:122] "RemoveContainer" containerID="9b0baeede1efc97f2c7c3af1e1183dee11ccc7faa1bd4e642ab895ff0f4b5a14" Oct 30 19:49:43.694104 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.694084 4433 scope.go:122] "RemoveContainer" containerID="9b0baeede1efc97f2c7c3af1e1183dee11ccc7faa1bd4e642ab895ff0f4b5a14" Oct 30 19:49:43.694585 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:49:43.694517 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b0baeede1efc97f2c7c3af1e1183dee11ccc7faa1bd4e642ab895ff0f4b5a14\": container with ID starting with 9b0baeede1efc97f2c7c3af1e1183dee11ccc7faa1bd4e642ab895ff0f4b5a14 not found: ID does not exist" containerID="9b0baeede1efc97f2c7c3af1e1183dee11ccc7faa1bd4e642ab895ff0f4b5a14" Oct 30 19:49:43.694585 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.694548 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b0baeede1efc97f2c7c3af1e1183dee11ccc7faa1bd4e642ab895ff0f4b5a14"} err="failed to get container status \"9b0baeede1efc97f2c7c3af1e1183dee11ccc7faa1bd4e642ab895ff0f4b5a14\": rpc error: code = NotFound desc = could not find container \"9b0baeede1efc97f2c7c3af1e1183dee11ccc7faa1bd4e642ab895ff0f4b5a14\": container with ID starting with 9b0baeede1efc97f2c7c3af1e1183dee11ccc7faa1bd4e642ab895ff0f4b5a14 not found: ID does not exist" Oct 30 19:49:43.742467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.742447 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" Oct 30 19:49:43.796536 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.796497 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b8ebe9-1b26-4870-b9ad-6535916b56f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b8ebe9-1b26-4870-b9ad-6535916b56f7-plugins-config-volume\") pod \"31b8ebe9-1b26-4870-b9ad-6535916b56f7\" (UID: \"31b8ebe9-1b26-4870-b9ad-6535916b56f7\") " Oct 30 19:49:43.796710 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.796573 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b8ebe9-1b26-4870-b9ad-6535916b56f7-kube-api-access-n7fvc\" (UniqueName: \"kubernetes.io/projected/31b8ebe9-1b26-4870-b9ad-6535916b56f7-kube-api-access-n7fvc\") pod \"31b8ebe9-1b26-4870-b9ad-6535916b56f7\" (UID: \"31b8ebe9-1b26-4870-b9ad-6535916b56f7\") " Oct 30 19:49:43.797052 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.796950 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b8ebe9-1b26-4870-b9ad-6535916b56f7-plugins-config-volume" pod "31b8ebe9-1b26-4870-b9ad-6535916b56f7" (UID: "31b8ebe9-1b26-4870-b9ad-6535916b56f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:43.797721 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.797699 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b8ebe9-1b26-4870-b9ad-6535916b56f7-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:43.800152 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.800125 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b8ebe9-1b26-4870-b9ad-6535916b56f7-kube-api-access-n7fvc" pod "31b8ebe9-1b26-4870-b9ad-6535916b56f7" (UID: "31b8ebe9-1b26-4870-b9ad-6535916b56f7"). InnerVolumeSpecName "kube-api-access-n7fvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:43.898755 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.898668 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/009b4710-bd3c-4273-bbee-84ebfbe13896-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009b4710-bd3c-4273-bbee-84ebfbe13896-plugins-config-volume\") pod \"009b4710-bd3c-4273-bbee-84ebfbe13896\" (UID: \"009b4710-bd3c-4273-bbee-84ebfbe13896\") " Oct 30 19:49:43.899076 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.899053 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/009b4710-bd3c-4273-bbee-84ebfbe13896-kube-api-access-vc7v8\" (UniqueName: \"kubernetes.io/projected/009b4710-bd3c-4273-bbee-84ebfbe13896-kube-api-access-vc7v8\") pod \"009b4710-bd3c-4273-bbee-84ebfbe13896\" (UID: \"009b4710-bd3c-4273-bbee-84ebfbe13896\") " Oct 30 19:49:43.899175 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.899114 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009b4710-bd3c-4273-bbee-84ebfbe13896-plugins-config-volume" pod "009b4710-bd3c-4273-bbee-84ebfbe13896" (UID: "009b4710-bd3c-4273-bbee-84ebfbe13896"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:43.899853 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.899828 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009b4710-bd3c-4273-bbee-84ebfbe13896-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:43.899967 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.899856 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7fvc\" (UniqueName: \"kubernetes.io/projected/31b8ebe9-1b26-4870-b9ad-6535916b56f7-kube-api-access-n7fvc\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:43.901828 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:43.901799 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009b4710-bd3c-4273-bbee-84ebfbe13896-kube-api-access-vc7v8" pod "009b4710-bd3c-4273-bbee-84ebfbe13896" (UID: "009b4710-bd3c-4273-bbee-84ebfbe13896"). InnerVolumeSpecName "kube-api-access-vc7v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:44.001786 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:44.001759 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc7v8\" (UniqueName: \"kubernetes.io/projected/009b4710-bd3c-4273-bbee-84ebfbe13896-kube-api-access-vc7v8\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:44.105667 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:44.105634 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7"] Oct 30 19:49:44.109193 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:44.109171 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df789477-58687686bb-nw8m7"] Oct 30 19:49:44.690610 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:44.690535 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" Oct 30 19:49:44.690610 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:44.690570 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp" event={"ID":"009b4710-bd3c-4273-bbee-84ebfbe13896","Type":"ContainerDied","Data":"f5e3eb92e04149c7097ad2b7f19a19b4d516c3fce64aa16d165bdf526e0c0a27"} Oct 30 19:49:44.691062 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:44.690622 4433 scope.go:122] "RemoveContainer" containerID="0a22cfe8e02833ea8b2cc4c52da3b725fd489bbb84a08dd75dec4469cdb193e0" Oct 30 19:49:44.856106 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:44.856072 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp"] Oct 30 19:49:44.859908 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:44.859885 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df789477-74d965dc9-jwhzp"] Oct 30 19:49:45.163392 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:45.163362 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:45.215838 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:45.215802 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="009b4710-bd3c-4273-bbee-84ebfbe13896" path="/var/lib/kubelet/pods/009b4710-bd3c-4273-bbee-84ebfbe13896/volumes" Oct 30 19:49:45.216396 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:45.216379 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b8ebe9-1b26-4870-b9ad-6535916b56f7" path="/var/lib/kubelet/pods/31b8ebe9-1b26-4870-b9ad-6535916b56f7/volumes" Oct 30 19:49:49.120008 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:49.117408 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n"] Oct 30 19:49:49.120008 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:49.117690 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" podUID="d340065c-df73-4f25-bcf9-2b984bdfd710" containerName="maas-api" containerID="cri-o://cf7d43dbdd9b11749b1672522f0676022eadfb7ecdeb64ea82b92e72d1cd9bba" gracePeriod=30 Oct 30 19:49:49.431059 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:49.413802 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:49.862626 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:49.862587 4433 generic.go:356] "Generic (PLEG): container finished" podID="d340065c-df73-4f25-bcf9-2b984bdfd710" containerID="cf7d43dbdd9b11749b1672522f0676022eadfb7ecdeb64ea82b92e72d1cd9bba" exitCode=0 Oct 30 19:49:49.863093 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:49.863064 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" event={"ID":"d340065c-df73-4f25-bcf9-2b984bdfd710","Type":"ContainerDied","Data":"cf7d43dbdd9b11749b1672522f0676022eadfb7ecdeb64ea82b92e72d1cd9bba"} Oct 30 19:49:50.610053 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:50.606645 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:49:50.816963 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:50.806721 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d340065c-df73-4f25-bcf9-2b984bdfd710-kube-api-access-696x4\" (UniqueName: \"kubernetes.io/projected/d340065c-df73-4f25-bcf9-2b984bdfd710-kube-api-access-696x4\") pod \"d340065c-df73-4f25-bcf9-2b984bdfd710\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " Oct 30 19:49:50.817296 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:50.817267 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-maas-api-tls\") pod \"d340065c-df73-4f25-bcf9-2b984bdfd710\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " Oct 30 19:49:50.817437 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:50.817423 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d340065c-df73-4f25-bcf9-2b984bdfd710-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d340065c-df73-4f25-bcf9-2b984bdfd710-openshift-service-ca\") pod \"d340065c-df73-4f25-bcf9-2b984bdfd710\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " Oct 30 19:49:50.914820 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:50.914729 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-metrics-certs\") pod \"d340065c-df73-4f25-bcf9-2b984bdfd710\" (UID: \"d340065c-df73-4f25-bcf9-2b984bdfd710\") " Oct 30 19:49:50.925026 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:50.886822 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" event={"ID":"d340065c-df73-4f25-bcf9-2b984bdfd710","Type":"ContainerDied","Data":"a14d7932bbc04a3d22d7fff8f768584bd6b6f1d4e8fa15c07f8cd426a046a8a4"} Oct 30 19:49:50.925374 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:50.925341 4433 scope.go:122] "RemoveContainer" containerID="cf7d43dbdd9b11749b1672522f0676022eadfb7ecdeb64ea82b92e72d1cd9bba" Oct 30 19:49:50.925727 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:50.828344 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d340065c-df73-4f25-bcf9-2b984bdfd710-openshift-service-ca" pod "d340065c-df73-4f25-bcf9-2b984bdfd710" (UID: "d340065c-df73-4f25-bcf9-2b984bdfd710"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:51.039671 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:50.968328 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gtk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:49:51.039671 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:51.035856 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:51.039671 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:50.886905 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n" Oct 30 19:49:51.039671 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:50.969289 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-metrics-certs" pod "d340065c-df73-4f25-bcf9-2b984bdfd710" (UID: "d340065c-df73-4f25-bcf9-2b984bdfd710"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:49:51.077001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:51.071504 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d340065c-df73-4f25-bcf9-2b984bdfd710-kube-api-access-696x4" pod "d340065c-df73-4f25-bcf9-2b984bdfd710" (UID: "d340065c-df73-4f25-bcf9-2b984bdfd710"). InnerVolumeSpecName "kube-api-access-696x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:51.128555 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:51.128483 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:51.140625 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:51.140588 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-696x4\" (UniqueName: \"kubernetes.io/projected/d340065c-df73-4f25-bcf9-2b984bdfd710-kube-api-access-696x4\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:51.140828 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:51.140808 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d340065c-df73-4f25-bcf9-2b984bdfd710-openshift-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:51.140953 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:51.140938 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-metrics-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:51.186108 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:51.186044 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-maas-api-tls" pod "d340065c-df73-4f25-bcf9-2b984bdfd710" (UID: "d340065c-df73-4f25-bcf9-2b984bdfd710"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:49:51.241854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:51.241826 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d340065c-df73-4f25-bcf9-2b984bdfd710-maas-api-tls\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:49:51.456461 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:51.456345 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n"] Oct 30 19:49:51.458621 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:51.458572 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:51.461655 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:51.461629 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8a3c7851-77ff8ddd5f-gzz7n"] Oct 30 19:49:51.701154 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:51.701064 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:51.945754 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:51.945720 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:52.213134 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:52.212448 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:52.658245 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:52.658210 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:53.218368 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:53.218296 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d340065c-df73-4f25-bcf9-2b984bdfd710" path="/var/lib/kubelet/pods/d340065c-df73-4f25-bcf9-2b984bdfd710/volumes" Oct 30 19:49:53.244037 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:53.244008 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:53.368165 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:49:53.368130 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:49:53.840273 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:53.840238 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:54.080312 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:54.080282 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:56.635413 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:56.635374 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:56.894454 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:56.894380 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:57.134264 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:49:57.134231 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:00.884104 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:00.884070 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz"] Oct 30 19:50:00.884519 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:00.884290 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" podUID="95740794-649d-4a88-9f4b-ada9e9fe7d2e" containerName="maas-api" containerID="cri-o://d036fecf0aaf0850d02f6e68b1cb6d156c68a20f540db6d8d791b39ed03c527a" gracePeriod=30 Oct 30 19:50:01.096269 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:01.087151 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:01.246328 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:01.246299 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:50:01.354589 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:01.354553 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-maas-api-tls\") pod \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " Oct 30 19:50:01.354791 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:01.354605 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95740794-649d-4a88-9f4b-ada9e9fe7d2e-kube-api-access-7f4fp\" (UniqueName: \"kubernetes.io/projected/95740794-649d-4a88-9f4b-ada9e9fe7d2e-kube-api-access-7f4fp\") pod \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " Oct 30 19:50:01.354791 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:01.354684 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95740794-649d-4a88-9f4b-ada9e9fe7d2e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95740794-649d-4a88-9f4b-ada9e9fe7d2e-openshift-service-ca\") pod \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " Oct 30 19:50:01.354946 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:01.354926 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-metrics-certs\") pod \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\" (UID: \"95740794-649d-4a88-9f4b-ada9e9fe7d2e\") " Oct 30 19:50:01.356963 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:01.356932 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95740794-649d-4a88-9f4b-ada9e9fe7d2e-openshift-service-ca" pod "95740794-649d-4a88-9f4b-ada9e9fe7d2e" (UID: "95740794-649d-4a88-9f4b-ada9e9fe7d2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:01.357515 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:01.357492 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95740794-649d-4a88-9f4b-ada9e9fe7d2e-openshift-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:01.358536 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:01.358502 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-maas-api-tls" pod "95740794-649d-4a88-9f4b-ada9e9fe7d2e" (UID: "95740794-649d-4a88-9f4b-ada9e9fe7d2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:01.358871 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:01.358843 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95740794-649d-4a88-9f4b-ada9e9fe7d2e-kube-api-access-7f4fp" pod "95740794-649d-4a88-9f4b-ada9e9fe7d2e" (UID: "95740794-649d-4a88-9f4b-ada9e9fe7d2e"). InnerVolumeSpecName "kube-api-access-7f4fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:01.396055 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:01.393132 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-metrics-certs" pod "95740794-649d-4a88-9f4b-ada9e9fe7d2e" (UID: "95740794-649d-4a88-9f4b-ada9e9fe7d2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:01.460039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:01.459686 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-metrics-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:01.460039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:01.459723 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95740794-649d-4a88-9f4b-ada9e9fe7d2e-maas-api-tls\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:01.460039 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:01.459740 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f4fp\" (UniqueName: \"kubernetes.io/projected/95740794-649d-4a88-9f4b-ada9e9fe7d2e-kube-api-access-7f4fp\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:02.006611 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:02.006572 4433 generic.go:356] "Generic (PLEG): container finished" podID="95740794-649d-4a88-9f4b-ada9e9fe7d2e" containerID="d036fecf0aaf0850d02f6e68b1cb6d156c68a20f540db6d8d791b39ed03c527a" exitCode=0 Oct 30 19:50:02.007443 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:02.007415 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" event={"ID":"95740794-649d-4a88-9f4b-ada9e9fe7d2e","Type":"ContainerDied","Data":"d036fecf0aaf0850d02f6e68b1cb6d156c68a20f540db6d8d791b39ed03c527a"} Oct 30 19:50:02.007711 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:02.007676 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" event={"ID":"95740794-649d-4a88-9f4b-ada9e9fe7d2e","Type":"ContainerDied","Data":"48d5b6c7868874681ea14bbe8a6b6e078b5b5cd72a4a23217c0c3a2c891db879"} Oct 30 19:50:02.007804 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:02.007720 4433 scope.go:122] "RemoveContainer" containerID="d036fecf0aaf0850d02f6e68b1cb6d156c68a20f540db6d8d791b39ed03c527a" Oct 30 19:50:02.007872 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:02.007468 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz" Oct 30 19:50:02.021701 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:02.021570 4433 scope.go:122] "RemoveContainer" containerID="d036fecf0aaf0850d02f6e68b1cb6d156c68a20f540db6d8d791b39ed03c527a" Oct 30 19:50:02.021996 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:50:02.021955 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d036fecf0aaf0850d02f6e68b1cb6d156c68a20f540db6d8d791b39ed03c527a\": container with ID starting with d036fecf0aaf0850d02f6e68b1cb6d156c68a20f540db6d8d791b39ed03c527a not found: ID does not exist" containerID="d036fecf0aaf0850d02f6e68b1cb6d156c68a20f540db6d8d791b39ed03c527a" Oct 30 19:50:02.022117 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:02.022058 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d036fecf0aaf0850d02f6e68b1cb6d156c68a20f540db6d8d791b39ed03c527a"} err="failed to get container status \"d036fecf0aaf0850d02f6e68b1cb6d156c68a20f540db6d8d791b39ed03c527a\": rpc error: code = NotFound desc = could not find container \"d036fecf0aaf0850d02f6e68b1cb6d156c68a20f540db6d8d791b39ed03c527a\": container with ID starting with d036fecf0aaf0850d02f6e68b1cb6d156c68a20f540db6d8d791b39ed03c527a not found: ID does not exist" Oct 30 19:50:02.176956 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:02.176922 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz"] Oct 30 19:50:02.227918 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:02.227879 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df789477-6fd78f4b6b-6qvxz"] Oct 30 19:50:02.589706 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:02.589667 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:02.841123 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:02.836569 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:03.135266 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:03.132930 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:03.247494 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:03.247458 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95740794-649d-4a88-9f4b-ada9e9fe7d2e" path="/var/lib/kubelet/pods/95740794-649d-4a88-9f4b-ada9e9fe7d2e/volumes" Oct 30 19:50:03.360528 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:03.360490 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:04.999623 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:04.999562 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:05.409239 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:05.409206 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn"] Oct 30 19:50:05.409661 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:05.409600 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" podUID="2a62b512-6b1a-4974-8512-eb702c044ecd" containerName="istio-proxy" containerID="cri-o://5d5c9928154f9ec145404b27612622513a1a6f974819ec0073635822c452e4c1" gracePeriod=30 Oct 30 19:50:06.022162 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:06.022113 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:07.847365 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:07.847334 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:08.087910 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:08.087872 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:08.320779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:08.320748 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:09.134433 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:09.134393 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:10.684745 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.684720 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:50:10.779811 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.779723 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-envoy\") pod \"2a62b512-6b1a-4974-8512-eb702c044ecd\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " Oct 30 19:50:10.779969 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.779909 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a62b512-6b1a-4974-8512-eb702c044ecd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a62b512-6b1a-4974-8512-eb702c044ecd-istiod-ca-cert\") pod \"2a62b512-6b1a-4974-8512-eb702c044ecd\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " Oct 30 19:50:10.779969 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.779946 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a62b512-6b1a-4974-8512-eb702c044ecd-kube-api-access-tqbps\" (UniqueName: \"kubernetes.io/projected/2a62b512-6b1a-4974-8512-eb702c044ecd-kube-api-access-tqbps\") pod \"2a62b512-6b1a-4974-8512-eb702c044ecd\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " Oct 30 19:50:10.780136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.780028 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-podinfo\") pod \"2a62b512-6b1a-4974-8512-eb702c044ecd\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " Oct 30 19:50:10.780136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.780104 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-credential-socket\") pod \"2a62b512-6b1a-4974-8512-eb702c044ecd\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " Oct 30 19:50:10.780234 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.780163 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-workload-socket\") pod \"2a62b512-6b1a-4974-8512-eb702c044ecd\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " Oct 30 19:50:10.780234 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.780205 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-workload-certs\") pod \"2a62b512-6b1a-4974-8512-eb702c044ecd\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " Oct 30 19:50:10.780340 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.780261 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-data\") pod \"2a62b512-6b1a-4974-8512-eb702c044ecd\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " Oct 30 19:50:10.780340 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.780304 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-token\" (UniqueName: \"kubernetes.io/projected/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-token\") pod \"2a62b512-6b1a-4974-8512-eb702c044ecd\" (UID: \"2a62b512-6b1a-4974-8512-eb702c044ecd\") " Oct 30 19:50:10.780552 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.780445 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a62b512-6b1a-4974-8512-eb702c044ecd-istiod-ca-cert" pod "2a62b512-6b1a-4974-8512-eb702c044ecd" (UID: "2a62b512-6b1a-4974-8512-eb702c044ecd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:10.780685 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.780466 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-workload-socket" pod "2a62b512-6b1a-4974-8512-eb702c044ecd" (UID: "2a62b512-6b1a-4974-8512-eb702c044ecd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:10.780685 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.780514 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-credential-socket" pod "2a62b512-6b1a-4974-8512-eb702c044ecd" (UID: "2a62b512-6b1a-4974-8512-eb702c044ecd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:10.780814 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.780684 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-workload-certs" pod "2a62b512-6b1a-4974-8512-eb702c044ecd" (UID: "2a62b512-6b1a-4974-8512-eb702c044ecd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:10.780814 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.780695 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-data" pod "2a62b512-6b1a-4974-8512-eb702c044ecd" (UID: "2a62b512-6b1a-4974-8512-eb702c044ecd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:10.781136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.781114 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:10.781222 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.781138 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:10.781222 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.781152 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:10.781222 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.781164 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a62b512-6b1a-4974-8512-eb702c044ecd-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:10.781222 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.781178 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:10.782332 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.782307 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a62b512-6b1a-4974-8512-eb702c044ecd-kube-api-access-tqbps" pod "2a62b512-6b1a-4974-8512-eb702c044ecd" (UID: "2a62b512-6b1a-4974-8512-eb702c044ecd"). InnerVolumeSpecName "kube-api-access-tqbps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:10.782481 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.782450 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-token" pod "2a62b512-6b1a-4974-8512-eb702c044ecd" (UID: "2a62b512-6b1a-4974-8512-eb702c044ecd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:10.782603 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.782496 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-envoy" pod "2a62b512-6b1a-4974-8512-eb702c044ecd" (UID: "2a62b512-6b1a-4974-8512-eb702c044ecd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:10.782603 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.782543 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-podinfo" pod "2a62b512-6b1a-4974-8512-eb702c044ecd" (UID: "2a62b512-6b1a-4974-8512-eb702c044ecd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:50:10.883596 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.883557 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:10.883596 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.883596 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqbps\" (UniqueName: \"kubernetes.io/projected/2a62b512-6b1a-4974-8512-eb702c044ecd-kube-api-access-tqbps\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:10.883846 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.883610 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:10.883846 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:10.883623 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a62b512-6b1a-4974-8512-eb702c044ecd-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:50:11.151043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:11.150974 4433 generic.go:356] "Generic (PLEG): container finished" podID="2a62b512-6b1a-4974-8512-eb702c044ecd" containerID="5d5c9928154f9ec145404b27612622513a1a6f974819ec0073635822c452e4c1" exitCode=0 Oct 30 19:50:11.151043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:11.151027 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" event={"ID":"2a62b512-6b1a-4974-8512-eb702c044ecd","Type":"ContainerDied","Data":"5d5c9928154f9ec145404b27612622513a1a6f974819ec0073635822c452e4c1"} Oct 30 19:50:11.151268 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:11.151052 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" Oct 30 19:50:11.151268 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:11.151113 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn" event={"ID":"2a62b512-6b1a-4974-8512-eb702c044ecd","Type":"ContainerDied","Data":"2e7111d135e462529dd431a8d9f81660aa83dccbb746bec6d3620a62bbe68409"} Oct 30 19:50:11.151268 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:11.151137 4433 scope.go:122] "RemoveContainer" containerID="5d5c9928154f9ec145404b27612622513a1a6f974819ec0073635822c452e4c1" Oct 30 19:50:11.162456 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:11.162436 4433 scope.go:122] "RemoveContainer" containerID="5d5c9928154f9ec145404b27612622513a1a6f974819ec0073635822c452e4c1" Oct 30 19:50:11.162721 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:50:11.162701 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5c9928154f9ec145404b27612622513a1a6f974819ec0073635822c452e4c1\": container with ID starting with 5d5c9928154f9ec145404b27612622513a1a6f974819ec0073635822c452e4c1 not found: ID does not exist" containerID="5d5c9928154f9ec145404b27612622513a1a6f974819ec0073635822c452e4c1" Oct 30 19:50:11.162798 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:11.162729 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5c9928154f9ec145404b27612622513a1a6f974819ec0073635822c452e4c1"} err="failed to get container status \"5d5c9928154f9ec145404b27612622513a1a6f974819ec0073635822c452e4c1\": rpc error: code = NotFound desc = could not find container \"5d5c9928154f9ec145404b27612622513a1a6f974819ec0073635822c452e4c1\": container with ID starting with 5d5c9928154f9ec145404b27612622513a1a6f974819ec0073635822c452e4c1 not found: ID does not exist" Oct 30 19:50:11.316454 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:11.316417 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn"] Oct 30 19:50:11.365970 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:11.365933 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-j6jwn"] Oct 30 19:50:13.079613 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:13.079581 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:13.217016 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:13.216966 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a62b512-6b1a-4974-8512-eb702c044ecd" path="/var/lib/kubelet/pods/2a62b512-6b1a-4974-8512-eb702c044ecd/volumes" Oct 30 19:50:13.500758 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:13.500677 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:15.642171 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:15.642135 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:15.868389 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:15.868361 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:16.367343 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:16.367302 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:16.655789 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:16.655685 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:16.953825 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:16.953731 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:17.949809 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:17.949769 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:21.069259 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:21.069221 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:21.326437 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:21.326355 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:22.622573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:22.622538 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:22.883326 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:22.883239 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:23.368858 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:50:23.368823 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:50:24.436460 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:24.436423 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:24.667328 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:24.667285 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:27.784345 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:27.784305 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:30.067921 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:30.067873 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:31.729591 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:31.729558 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:32.147161 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:32.147124 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:32.640300 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:32.640266 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:32.900580 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:32.900471 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:33.319346 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:33.319298 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:34.695045 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:34.695008 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:37.483424 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:37.483378 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:39.606204 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:39.606155 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:40.648754 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:40.648716 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:41.087400 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:41.087371 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:42.220618 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:42.220561 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:45.146619 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:45.146583 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:45.385266 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:45.385227 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:47.251353 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:47.251317 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:48.535312 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:48.535233 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:48.773999 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:48.770641 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:52.055645 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:52.055604 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:53.367190 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:50:53.367149 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:50:55.187169 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:55.187137 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:55.808138 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:55.808096 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:56.065824 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:56.065791 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:56.685572 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:50:56.685538 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:01.299091 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:01.299055 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:05.905392 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:05.905318 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:06.133495 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:06.133463 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:06.846329 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:06.846301 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:07.173923 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:07.173855 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:10.875626 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:10.875522 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:13.844138 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:13.844102 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:15.605930 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:15.605897 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:20.303397 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:20.303274 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:20.529704 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:20.529664 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:20.743932 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:20.743892 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:23.404332 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:51:23.404263 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:51:25.182153 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:25.182111 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:28.558629 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:28.558598 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:29.867639 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:29.867578 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:32.209612 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:32.209573 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:32.946185 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:32.946149 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:34.397885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:34.397845 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:34.670833 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:34.670732 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:34.916730 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:34.916698 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:35.221013 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:35.220948 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:39.223031 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:39.222951 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:39.445241 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:39.445210 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:44.048973 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:44.048885 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:48.853741 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:48.853690 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:51.024960 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:51.024927 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm"] Oct 30 19:51:51.025377 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:51.025184 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" podUID="92e60493-90ad-4718-a5a5-9acb15f7af19" containerName="istio-proxy" containerID="cri-o://7f630d4df21927d285584e21cadf7fa0ea9fe4d58466f8972b8e37dd8a97eeeb" gracePeriod=30 Oct 30 19:51:53.366810 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:51:53.366773 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:51:53.465076 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:53.465045 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:55.144388 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:55.144347 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:51:55.144921 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:55.144416 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" podUID="92e60493-90ad-4718-a5a5-9acb15f7af19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:51:55.937863 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:55.937831 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb"] Oct 30 19:51:55.938109 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:55.938084 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" podUID="53a6ee20-cbc7-49e8-b581-19310616c0ee" containerName="istio-proxy" containerID="cri-o://a377e6c5b990a9317e05e0fc45253e70ecc73cedf7d46e5237fb6ab1e96d65e3" gracePeriod=30 Oct 30 19:51:56.295129 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.295105 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:51:56.453187 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.453102 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-token\" (UniqueName: \"kubernetes.io/projected/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-token\") pod \"92e60493-90ad-4718-a5a5-9acb15f7af19\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " Oct 30 19:51:56.453364 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.453241 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-workload-certs\") pod \"92e60493-90ad-4718-a5a5-9acb15f7af19\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " Oct 30 19:51:56.453364 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.453289 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e60493-90ad-4718-a5a5-9acb15f7af19-kube-api-access-h4nlm\" (UniqueName: \"kubernetes.io/projected/92e60493-90ad-4718-a5a5-9acb15f7af19-kube-api-access-h4nlm\") pod \"92e60493-90ad-4718-a5a5-9acb15f7af19\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " Oct 30 19:51:56.453489 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.453365 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-credential-socket\") pod \"92e60493-90ad-4718-a5a5-9acb15f7af19\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " Oct 30 19:51:56.453489 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.453462 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e60493-90ad-4718-a5a5-9acb15f7af19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e60493-90ad-4718-a5a5-9acb15f7af19-istiod-ca-cert\") pod \"92e60493-90ad-4718-a5a5-9acb15f7af19\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " Oct 30 19:51:56.453596 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.453511 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-podinfo\") pod \"92e60493-90ad-4718-a5a5-9acb15f7af19\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " Oct 30 19:51:56.453651 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.453598 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-workload-certs" pod "92e60493-90ad-4718-a5a5-9acb15f7af19" (UID: "92e60493-90ad-4718-a5a5-9acb15f7af19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:56.453706 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.453669 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-workload-socket\") pod \"92e60493-90ad-4718-a5a5-9acb15f7af19\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " Oct 30 19:51:56.453756 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.453728 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-envoy\") pod \"92e60493-90ad-4718-a5a5-9acb15f7af19\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " Oct 30 19:51:56.454006 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.453821 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-data\") pod \"92e60493-90ad-4718-a5a5-9acb15f7af19\" (UID: \"92e60493-90ad-4718-a5a5-9acb15f7af19\") " Oct 30 19:51:56.454006 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.453877 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e60493-90ad-4718-a5a5-9acb15f7af19-istiod-ca-cert" pod "92e60493-90ad-4718-a5a5-9acb15f7af19" (UID: "92e60493-90ad-4718-a5a5-9acb15f7af19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:56.454257 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.454223 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-workload-socket" pod "92e60493-90ad-4718-a5a5-9acb15f7af19" (UID: "92e60493-90ad-4718-a5a5-9acb15f7af19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:56.454429 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.454397 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-credential-socket" pod "92e60493-90ad-4718-a5a5-9acb15f7af19" (UID: "92e60493-90ad-4718-a5a5-9acb15f7af19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:56.454566 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.454444 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-data" pod "92e60493-90ad-4718-a5a5-9acb15f7af19" (UID: "92e60493-90ad-4718-a5a5-9acb15f7af19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:56.454685 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.454609 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:51:56.454789 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.454688 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:51:56.454789 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.454704 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e60493-90ad-4718-a5a5-9acb15f7af19-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:51:56.456517 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.456483 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-podinfo" pod "92e60493-90ad-4718-a5a5-9acb15f7af19" (UID: "92e60493-90ad-4718-a5a5-9acb15f7af19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:51:56.456721 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.456695 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-envoy" pod "92e60493-90ad-4718-a5a5-9acb15f7af19" (UID: "92e60493-90ad-4718-a5a5-9acb15f7af19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:56.457672 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.457652 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-token" pod "92e60493-90ad-4718-a5a5-9acb15f7af19" (UID: "92e60493-90ad-4718-a5a5-9acb15f7af19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:56.457894 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.457874 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e60493-90ad-4718-a5a5-9acb15f7af19-kube-api-access-h4nlm" pod "92e60493-90ad-4718-a5a5-9acb15f7af19" (UID: "92e60493-90ad-4718-a5a5-9acb15f7af19"). InnerVolumeSpecName "kube-api-access-h4nlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:56.555996 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.555952 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:51:56.556163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.556029 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:51:56.556163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.556045 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:51:56.556163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.556058 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4nlm\" (UniqueName: \"kubernetes.io/projected/92e60493-90ad-4718-a5a5-9acb15f7af19-kube-api-access-h4nlm\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:51:56.556163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.556071 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e60493-90ad-4718-a5a5-9acb15f7af19-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:51:56.556163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.556082 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e60493-90ad-4718-a5a5-9acb15f7af19-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:51:56.935870 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.935833 4433 generic.go:356] "Generic (PLEG): container finished" podID="92e60493-90ad-4718-a5a5-9acb15f7af19" containerID="7f630d4df21927d285584e21cadf7fa0ea9fe4d58466f8972b8e37dd8a97eeeb" exitCode=0 Oct 30 19:51:56.936067 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.935885 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" Oct 30 19:51:56.936067 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.935918 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" event={"ID":"92e60493-90ad-4718-a5a5-9acb15f7af19","Type":"ContainerDied","Data":"7f630d4df21927d285584e21cadf7fa0ea9fe4d58466f8972b8e37dd8a97eeeb"} Oct 30 19:51:56.936067 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.935989 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm" event={"ID":"92e60493-90ad-4718-a5a5-9acb15f7af19","Type":"ContainerDied","Data":"c0fc33e4c6dead84bce80ced69441389caecadf2c053aab3deb9af383b030da5"} Oct 30 19:51:56.936067 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.936015 4433 scope.go:122] "RemoveContainer" containerID="7f630d4df21927d285584e21cadf7fa0ea9fe4d58466f8972b8e37dd8a97eeeb" Oct 30 19:51:56.947667 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.947446 4433 scope.go:122] "RemoveContainer" containerID="7f630d4df21927d285584e21cadf7fa0ea9fe4d58466f8972b8e37dd8a97eeeb" Oct 30 19:51:56.947734 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:51:56.947722 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f630d4df21927d285584e21cadf7fa0ea9fe4d58466f8972b8e37dd8a97eeeb\": container with ID starting with 7f630d4df21927d285584e21cadf7fa0ea9fe4d58466f8972b8e37dd8a97eeeb not found: ID does not exist" containerID="7f630d4df21927d285584e21cadf7fa0ea9fe4d58466f8972b8e37dd8a97eeeb" Oct 30 19:51:56.947792 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:56.947744 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f630d4df21927d285584e21cadf7fa0ea9fe4d58466f8972b8e37dd8a97eeeb"} err="failed to get container status \"7f630d4df21927d285584e21cadf7fa0ea9fe4d58466f8972b8e37dd8a97eeeb\": rpc error: code = NotFound desc = could not find container \"7f630d4df21927d285584e21cadf7fa0ea9fe4d58466f8972b8e37dd8a97eeeb\": container with ID starting with 7f630d4df21927d285584e21cadf7fa0ea9fe4d58466f8972b8e37dd8a97eeeb not found: ID does not exist" Oct 30 19:51:57.104268 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:57.104240 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm"] Oct 30 19:51:57.153501 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:57.153472 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-sgtxm"] Oct 30 19:51:57.215868 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:57.215802 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e60493-90ad-4718-a5a5-9acb15f7af19" path="/var/lib/kubelet/pods/92e60493-90ad-4718-a5a5-9acb15f7af19/volumes" Oct 30 19:51:58.067462 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:58.067432 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:58.327047 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:58.326955 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:58.528256 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:58.528222 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr"] Oct 30 19:51:58.528431 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:58.528410 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" podUID="2583f5a9-8bc2-45f0-9470-418c96a00d14" containerName="istio-proxy" containerID="cri-o://eaa19f3fbc87685d8e96fe26d9edab6acbf2ec034e11a43cdcf7db6b9881b9ed" gracePeriod=30 Oct 30 19:51:59.185588 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:51:59.185541 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:01.205884 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.205824 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:52:01.313183 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.313155 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-envoy\") pod \"53a6ee20-cbc7-49e8-b581-19310616c0ee\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " Oct 30 19:52:01.313320 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.313233 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-workload-socket\") pod \"53a6ee20-cbc7-49e8-b581-19310616c0ee\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " Oct 30 19:52:01.313320 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.313259 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a6ee20-cbc7-49e8-b581-19310616c0ee-kube-api-access-97cp8\" (UniqueName: \"kubernetes.io/projected/53a6ee20-cbc7-49e8-b581-19310616c0ee-kube-api-access-97cp8\") pod \"53a6ee20-cbc7-49e8-b581-19310616c0ee\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " Oct 30 19:52:01.313401 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.313324 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-podinfo\") pod \"53a6ee20-cbc7-49e8-b581-19310616c0ee\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " Oct 30 19:52:01.313401 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.313340 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-credential-socket\") pod \"53a6ee20-cbc7-49e8-b581-19310616c0ee\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " Oct 30 19:52:01.313401 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.313376 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-workload-certs\") pod \"53a6ee20-cbc7-49e8-b581-19310616c0ee\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " Oct 30 19:52:01.313546 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.313498 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-data\") pod \"53a6ee20-cbc7-49e8-b581-19310616c0ee\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " Oct 30 19:52:01.313546 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.313515 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-workload-socket" pod "53a6ee20-cbc7-49e8-b581-19310616c0ee" (UID: "53a6ee20-cbc7-49e8-b581-19310616c0ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:01.313546 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.313540 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-token\" (UniqueName: \"kubernetes.io/projected/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-token\") pod \"53a6ee20-cbc7-49e8-b581-19310616c0ee\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " Oct 30 19:52:01.313696 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.313596 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53a6ee20-cbc7-49e8-b581-19310616c0ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a6ee20-cbc7-49e8-b581-19310616c0ee-istiod-ca-cert\") pod \"53a6ee20-cbc7-49e8-b581-19310616c0ee\" (UID: \"53a6ee20-cbc7-49e8-b581-19310616c0ee\") " Oct 30 19:52:01.313757 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.313718 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-workload-certs" pod "53a6ee20-cbc7-49e8-b581-19310616c0ee" (UID: "53a6ee20-cbc7-49e8-b581-19310616c0ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:01.313757 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.313730 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-data" pod "53a6ee20-cbc7-49e8-b581-19310616c0ee" (UID: "53a6ee20-cbc7-49e8-b581-19310616c0ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:01.314342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.314037 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-credential-socket" pod "53a6ee20-cbc7-49e8-b581-19310616c0ee" (UID: "53a6ee20-cbc7-49e8-b581-19310616c0ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:01.314342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.314167 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a6ee20-cbc7-49e8-b581-19310616c0ee-istiod-ca-cert" pod "53a6ee20-cbc7-49e8-b581-19310616c0ee" (UID: "53a6ee20-cbc7-49e8-b581-19310616c0ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:52:01.314342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.314319 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:01.314342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.314341 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:01.314619 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.314357 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:01.314619 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.314371 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:01.314619 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.314385 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a6ee20-cbc7-49e8-b581-19310616c0ee-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:01.315635 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.315614 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-podinfo" pod "53a6ee20-cbc7-49e8-b581-19310616c0ee" (UID: "53a6ee20-cbc7-49e8-b581-19310616c0ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:52:01.315832 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.315804 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-token" pod "53a6ee20-cbc7-49e8-b581-19310616c0ee" (UID: "53a6ee20-cbc7-49e8-b581-19310616c0ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:01.315901 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.315813 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-envoy" pod "53a6ee20-cbc7-49e8-b581-19310616c0ee" (UID: "53a6ee20-cbc7-49e8-b581-19310616c0ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:01.315901 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.315817 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a6ee20-cbc7-49e8-b581-19310616c0ee-kube-api-access-97cp8" pod "53a6ee20-cbc7-49e8-b581-19310616c0ee" (UID: "53a6ee20-cbc7-49e8-b581-19310616c0ee"). InnerVolumeSpecName "kube-api-access-97cp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:01.416035 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.416001 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:01.416035 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.416037 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:01.416212 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.416053 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97cp8\" (UniqueName: \"kubernetes.io/projected/53a6ee20-cbc7-49e8-b581-19310616c0ee-kube-api-access-97cp8\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:01.416212 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.416067 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a6ee20-cbc7-49e8-b581-19310616c0ee-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:01.972363 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.972328 4433 generic.go:356] "Generic (PLEG): container finished" podID="53a6ee20-cbc7-49e8-b581-19310616c0ee" containerID="a377e6c5b990a9317e05e0fc45253e70ecc73cedf7d46e5237fb6ab1e96d65e3" exitCode=0 Oct 30 19:52:01.972625 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.972376 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" Oct 30 19:52:01.972625 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.972384 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" event={"ID":"53a6ee20-cbc7-49e8-b581-19310616c0ee","Type":"ContainerDied","Data":"a377e6c5b990a9317e05e0fc45253e70ecc73cedf7d46e5237fb6ab1e96d65e3"} Oct 30 19:52:01.972625 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.972498 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb" event={"ID":"53a6ee20-cbc7-49e8-b581-19310616c0ee","Type":"ContainerDied","Data":"2921c5c72cc04bb01b18427ba341b58a10b648384d44bdab53de0cf484ca76bf"} Oct 30 19:52:01.972625 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.972524 4433 scope.go:122] "RemoveContainer" containerID="a377e6c5b990a9317e05e0fc45253e70ecc73cedf7d46e5237fb6ab1e96d65e3" Oct 30 19:52:01.983537 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.983517 4433 scope.go:122] "RemoveContainer" containerID="a377e6c5b990a9317e05e0fc45253e70ecc73cedf7d46e5237fb6ab1e96d65e3" Oct 30 19:52:01.983793 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:52:01.983775 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a377e6c5b990a9317e05e0fc45253e70ecc73cedf7d46e5237fb6ab1e96d65e3\": container with ID starting with a377e6c5b990a9317e05e0fc45253e70ecc73cedf7d46e5237fb6ab1e96d65e3 not found: ID does not exist" containerID="a377e6c5b990a9317e05e0fc45253e70ecc73cedf7d46e5237fb6ab1e96d65e3" Oct 30 19:52:01.983831 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:01.983801 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a377e6c5b990a9317e05e0fc45253e70ecc73cedf7d46e5237fb6ab1e96d65e3"} err="failed to get container status \"a377e6c5b990a9317e05e0fc45253e70ecc73cedf7d46e5237fb6ab1e96d65e3\": rpc error: code = NotFound desc = could not find container \"a377e6c5b990a9317e05e0fc45253e70ecc73cedf7d46e5237fb6ab1e96d65e3\": container with ID starting with a377e6c5b990a9317e05e0fc45253e70ecc73cedf7d46e5237fb6ab1e96d65e3 not found: ID does not exist" Oct 30 19:52:02.186955 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:02.186931 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb"] Oct 30 19:52:02.239684 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:02.239614 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-qczxb"] Oct 30 19:52:02.892555 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:02.892516 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:03.214992 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.214909 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53a6ee20-cbc7-49e8-b581-19310616c0ee" path="/var/lib/kubelet/pods/53a6ee20-cbc7-49e8-b581-19310616c0ee/volumes" Oct 30 19:52:03.509639 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.509565 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:52:03.510006 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.509640 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" podUID="2583f5a9-8bc2-45f0-9470-418c96a00d14" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:52:03.804756 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.804725 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:52:03.841404 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.841375 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-credential-socket\") pod \"2583f5a9-8bc2-45f0-9470-418c96a00d14\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " Oct 30 19:52:03.841541 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.841515 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-token\" (UniqueName: \"kubernetes.io/projected/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-token\") pod \"2583f5a9-8bc2-45f0-9470-418c96a00d14\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " Oct 30 19:52:03.841640 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.841625 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-podinfo\") pod \"2583f5a9-8bc2-45f0-9470-418c96a00d14\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " Oct 30 19:52:03.841698 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.841624 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-credential-socket" pod "2583f5a9-8bc2-45f0-9470-418c96a00d14" (UID: "2583f5a9-8bc2-45f0-9470-418c96a00d14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:03.841758 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.841704 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2583f5a9-8bc2-45f0-9470-418c96a00d14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2583f5a9-8bc2-45f0-9470-418c96a00d14-istiod-ca-cert\") pod \"2583f5a9-8bc2-45f0-9470-418c96a00d14\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " Oct 30 19:52:03.841758 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.841748 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-workload-certs\") pod \"2583f5a9-8bc2-45f0-9470-418c96a00d14\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " Oct 30 19:52:03.841886 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.841870 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2583f5a9-8bc2-45f0-9470-418c96a00d14-kube-api-access-4hgkm\" (UniqueName: \"kubernetes.io/projected/2583f5a9-8bc2-45f0-9470-418c96a00d14-kube-api-access-4hgkm\") pod \"2583f5a9-8bc2-45f0-9470-418c96a00d14\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " Oct 30 19:52:03.842040 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.842020 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-envoy\") pod \"2583f5a9-8bc2-45f0-9470-418c96a00d14\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " Oct 30 19:52:03.842218 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.842097 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-workload-certs" pod "2583f5a9-8bc2-45f0-9470-418c96a00d14" (UID: "2583f5a9-8bc2-45f0-9470-418c96a00d14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:03.842315 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.842112 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2583f5a9-8bc2-45f0-9470-418c96a00d14-istiod-ca-cert" pod "2583f5a9-8bc2-45f0-9470-418c96a00d14" (UID: "2583f5a9-8bc2-45f0-9470-418c96a00d14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:52:03.842315 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.842194 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-workload-socket\") pod \"2583f5a9-8bc2-45f0-9470-418c96a00d14\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " Oct 30 19:52:03.842427 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.842311 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-data\") pod \"2583f5a9-8bc2-45f0-9470-418c96a00d14\" (UID: \"2583f5a9-8bc2-45f0-9470-418c96a00d14\") " Oct 30 19:52:03.842648 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.842596 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-data" pod "2583f5a9-8bc2-45f0-9470-418c96a00d14" (UID: "2583f5a9-8bc2-45f0-9470-418c96a00d14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:03.842800 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.842768 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-workload-socket" pod "2583f5a9-8bc2-45f0-9470-418c96a00d14" (UID: "2583f5a9-8bc2-45f0-9470-418c96a00d14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:03.843173 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.843151 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:03.843348 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.843289 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:03.843348 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.843316 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:03.843348 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.843333 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2583f5a9-8bc2-45f0-9470-418c96a00d14-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:03.843348 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.843348 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:03.845012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.844313 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-envoy" pod "2583f5a9-8bc2-45f0-9470-418c96a00d14" (UID: "2583f5a9-8bc2-45f0-9470-418c96a00d14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:03.845012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.844450 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2583f5a9-8bc2-45f0-9470-418c96a00d14-kube-api-access-4hgkm" pod "2583f5a9-8bc2-45f0-9470-418c96a00d14" (UID: "2583f5a9-8bc2-45f0-9470-418c96a00d14"). InnerVolumeSpecName "kube-api-access-4hgkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:03.845012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.844482 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-token" pod "2583f5a9-8bc2-45f0-9470-418c96a00d14" (UID: "2583f5a9-8bc2-45f0-9470-418c96a00d14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:03.845012 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.844540 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-podinfo" pod "2583f5a9-8bc2-45f0-9470-418c96a00d14" (UID: "2583f5a9-8bc2-45f0-9470-418c96a00d14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:52:03.945169 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.945144 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:03.945169 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.945169 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:03.945355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.945179 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hgkm\" (UniqueName: \"kubernetes.io/projected/2583f5a9-8bc2-45f0-9470-418c96a00d14-kube-api-access-4hgkm\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:03.945355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.945189 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2583f5a9-8bc2-45f0-9470-418c96a00d14-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:52:03.988264 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.988226 4433 generic.go:356] "Generic (PLEG): container finished" podID="2583f5a9-8bc2-45f0-9470-418c96a00d14" containerID="eaa19f3fbc87685d8e96fe26d9edab6acbf2ec034e11a43cdcf7db6b9881b9ed" exitCode=0 Oct 30 19:52:03.988415 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.988277 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" Oct 30 19:52:03.988415 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.988315 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" event={"ID":"2583f5a9-8bc2-45f0-9470-418c96a00d14","Type":"ContainerDied","Data":"eaa19f3fbc87685d8e96fe26d9edab6acbf2ec034e11a43cdcf7db6b9881b9ed"} Oct 30 19:52:03.988415 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.988378 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr" event={"ID":"2583f5a9-8bc2-45f0-9470-418c96a00d14","Type":"ContainerDied","Data":"b5b8850d687a2d616a2a0c18fd81811b3e398a2fde4d69ff502a1ee689ec5210"} Oct 30 19:52:03.988415 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:03.988406 4433 scope.go:122] "RemoveContainer" containerID="eaa19f3fbc87685d8e96fe26d9edab6acbf2ec034e11a43cdcf7db6b9881b9ed" Oct 30 19:52:04.002580 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:04.002555 4433 scope.go:122] "RemoveContainer" containerID="eaa19f3fbc87685d8e96fe26d9edab6acbf2ec034e11a43cdcf7db6b9881b9ed" Oct 30 19:52:04.002893 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:52:04.002872 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa19f3fbc87685d8e96fe26d9edab6acbf2ec034e11a43cdcf7db6b9881b9ed\": container with ID starting with eaa19f3fbc87685d8e96fe26d9edab6acbf2ec034e11a43cdcf7db6b9881b9ed not found: ID does not exist" containerID="eaa19f3fbc87685d8e96fe26d9edab6acbf2ec034e11a43cdcf7db6b9881b9ed" Oct 30 19:52:04.003037 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:04.002897 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa19f3fbc87685d8e96fe26d9edab6acbf2ec034e11a43cdcf7db6b9881b9ed"} err="failed to get container status \"eaa19f3fbc87685d8e96fe26d9edab6acbf2ec034e11a43cdcf7db6b9881b9ed\": rpc error: code = NotFound desc = could not find container \"eaa19f3fbc87685d8e96fe26d9edab6acbf2ec034e11a43cdcf7db6b9881b9ed\": container with ID starting with eaa19f3fbc87685d8e96fe26d9edab6acbf2ec034e11a43cdcf7db6b9881b9ed not found: ID does not exist" Oct 30 19:52:04.152894 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:04.152861 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr"] Oct 30 19:52:04.156136 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:04.156113 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-xxwrr"] Oct 30 19:52:05.215445 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:05.215416 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2583f5a9-8bc2-45f0-9470-418c96a00d14" path="/var/lib/kubelet/pods/2583f5a9-8bc2-45f0-9470-418c96a00d14/volumes" Oct 30 19:52:08.217572 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:08.217535 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:08.441236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:08.441204 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:11.836909 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:11.836862 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:12.143055 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:12.139999 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:13.043081 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:13.042962 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:14.221524 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:14.221485 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:15.422997 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:15.422560 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:15.648549 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:15.648514 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:18.202554 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:18.202520 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:22.957297 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:22.957255 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:23.366735 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:52:23.366687 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:52:28.080754 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:28.080716 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:32.544370 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:32.544339 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:32.868544 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:32.868501 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:37.908913 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:37.908840 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:40.401302 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:40.401269 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:41.154232 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:41.154198 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:42.978416 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:42.978374 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:43.238967 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:43.238876 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:43.513250 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:43.513165 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:45.512612 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:45.512553 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:47.651595 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:47.651514 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:47.877374 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:47.877327 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:48.102675 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:48.102634 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:52.999388 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:52.999357 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:53.368081 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:52:53.368047 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:52:57.203663 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:57.203630 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:57.507378 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:57.507256 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:58.105394 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:58.105360 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:59.025993 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:59.025955 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:59.990917 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:52:59.990885 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:00.230550 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:00.230520 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:01.900045 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:01.899963 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:02.147123 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:02.147067 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:03.004551 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:03.004480 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:04.450303 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:04.450259 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:04.691944 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:04.691909 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:06.517636 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:06.517600 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:07.782406 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:07.782367 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:08.059534 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:08.059451 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:08.311079 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:08.300529 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:12.436453 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:12.436405 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz"] Oct 30 19:53:12.456090 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:12.456020 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" podUID="3ac90de6-4478-4ba6-b106-d40c7c6be7b9" containerName="maas-api" containerID="cri-o://ac9ee728e6e09bcf8970690e13cecd4456fcca3ebe75afc379b1017b44feb37a" gracePeriod=30 Oct 30 19:53:12.539399 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:12.539350 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:12.942131 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:12.942102 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:53:12.964495 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:12.964449 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-openshift-service-ca\") pod \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " Oct 30 19:53:12.964716 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:12.964562 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-kube-api-access-h958h\" (UniqueName: \"kubernetes.io/projected/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-kube-api-access-h958h\") pod \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " Oct 30 19:53:12.964716 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:12.964702 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-metrics-certs\") pod \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " Oct 30 19:53:12.964883 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:12.964853 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-openshift-service-ca" pod "3ac90de6-4478-4ba6-b106-d40c7c6be7b9" (UID: "3ac90de6-4478-4ba6-b106-d40c7c6be7b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:12.965275 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:12.965255 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-openshift-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:53:12.966863 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:12.966837 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-metrics-certs" pod "3ac90de6-4478-4ba6-b106-d40c7c6be7b9" (UID: "3ac90de6-4478-4ba6-b106-d40c7c6be7b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:12.966974 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:12.966869 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-kube-api-access-h958h" pod "3ac90de6-4478-4ba6-b106-d40c7c6be7b9" (UID: "3ac90de6-4478-4ba6-b106-d40c7c6be7b9"). InnerVolumeSpecName "kube-api-access-h958h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:13.066760 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:13.066708 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-maas-api-tls\") pod \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\" (UID: \"3ac90de6-4478-4ba6-b106-d40c7c6be7b9\") " Oct 30 19:53:13.067609 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:13.067586 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h958h\" (UniqueName: \"kubernetes.io/projected/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-kube-api-access-h958h\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:53:13.067609 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:13.067611 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-metrics-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:53:13.068888 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:13.068863 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-maas-api-tls" pod "3ac90de6-4478-4ba6-b106-d40c7c6be7b9" (UID: "3ac90de6-4478-4ba6-b106-d40c7c6be7b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:13.169457 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:13.169363 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac90de6-4478-4ba6-b106-d40c7c6be7b9-maas-api-tls\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:53:13.570624 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:13.570591 4433 generic.go:356] "Generic (PLEG): container finished" podID="3ac90de6-4478-4ba6-b106-d40c7c6be7b9" containerID="ac9ee728e6e09bcf8970690e13cecd4456fcca3ebe75afc379b1017b44feb37a" exitCode=0 Oct 30 19:53:13.571123 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:13.570642 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" Oct 30 19:53:13.571123 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:13.570676 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" event={"ID":"3ac90de6-4478-4ba6-b106-d40c7c6be7b9","Type":"ContainerDied","Data":"ac9ee728e6e09bcf8970690e13cecd4456fcca3ebe75afc379b1017b44feb37a"} Oct 30 19:53:13.571123 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:13.570734 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz" event={"ID":"3ac90de6-4478-4ba6-b106-d40c7c6be7b9","Type":"ContainerDied","Data":"08b4cfedf08579bab064d922ca06c84e717622f32fc083156e7a534c860e693b"} Oct 30 19:53:13.571123 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:13.570761 4433 scope.go:122] "RemoveContainer" containerID="ac9ee728e6e09bcf8970690e13cecd4456fcca3ebe75afc379b1017b44feb37a" Oct 30 19:53:13.583308 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:13.583024 4433 scope.go:122] "RemoveContainer" containerID="ac9ee728e6e09bcf8970690e13cecd4456fcca3ebe75afc379b1017b44feb37a" Oct 30 19:53:13.583453 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:53:13.583428 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9ee728e6e09bcf8970690e13cecd4456fcca3ebe75afc379b1017b44feb37a\": container with ID starting with ac9ee728e6e09bcf8970690e13cecd4456fcca3ebe75afc379b1017b44feb37a not found: ID does not exist" containerID="ac9ee728e6e09bcf8970690e13cecd4456fcca3ebe75afc379b1017b44feb37a" Oct 30 19:53:13.583511 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:13.583467 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9ee728e6e09bcf8970690e13cecd4456fcca3ebe75afc379b1017b44feb37a"} err="failed to get container status \"ac9ee728e6e09bcf8970690e13cecd4456fcca3ebe75afc379b1017b44feb37a\": rpc error: code = NotFound desc = could not find container \"ac9ee728e6e09bcf8970690e13cecd4456fcca3ebe75afc379b1017b44feb37a\": container with ID starting with ac9ee728e6e09bcf8970690e13cecd4456fcca3ebe75afc379b1017b44feb37a not found: ID does not exist" Oct 30 19:53:13.734802 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:13.734755 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz"] Oct 30 19:53:13.783778 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:13.783692 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5bf13ff3-58765bb45-cj4qz"] Oct 30 19:53:14.159481 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:14.159446 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:14.393644 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:14.393605 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:14.632507 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:14.632468 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:14.965826 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:14.965717 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:15.199853 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.199816 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:15.215560 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.215526 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac90de6-4478-4ba6-b106-d40c7c6be7b9" path="/var/lib/kubelet/pods/3ac90de6-4478-4ba6-b106-d40c7c6be7b9/volumes" Oct 30 19:53:15.439056 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.439020 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:15.743099 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.731674 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd"] Oct 30 19:53:15.743099 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.732903 4433 cpu_manager.go:414] "containerMap: removing container" podUID="2a62b512-6b1a-4974-8512-eb702c044ecd" containerName="istio-proxy" Oct 30 19:53:15.743099 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.732938 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a62b512-6b1a-4974-8512-eb702c044ecd" containerName="istio-proxy" Oct 30 19:53:15.743099 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.732958 4433 cpu_manager.go:414] "containerMap: removing container" podUID="009b4710-bd3c-4273-bbee-84ebfbe13896" containerName="payload-pre-processing" Oct 30 19:53:15.743099 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.732965 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="009b4710-bd3c-4273-bbee-84ebfbe13896" containerName="payload-pre-processing" Oct 30 19:53:15.743099 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.733010 4433 cpu_manager.go:414] "containerMap: removing container" podUID="92e60493-90ad-4718-a5a5-9acb15f7af19" containerName="istio-proxy" Oct 30 19:53:15.743099 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.733018 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e60493-90ad-4718-a5a5-9acb15f7af19" containerName="istio-proxy" Oct 30 19:53:15.743099 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.733031 4433 cpu_manager.go:414] "containerMap: removing container" podUID="07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" containerName="istio-proxy" Oct 30 19:53:15.743099 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.733038 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" containerName="istio-proxy" Oct 30 19:53:15.743099 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.733058 4433 cpu_manager.go:414] "containerMap: removing container" podUID="2583f5a9-8bc2-45f0-9470-418c96a00d14" containerName="istio-proxy" Oct 30 19:53:15.743099 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.733066 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2583f5a9-8bc2-45f0-9470-418c96a00d14" containerName="istio-proxy" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743138 4433 cpu_manager.go:414] "containerMap: removing container" podUID="53a6ee20-cbc7-49e8-b581-19310616c0ee" containerName="istio-proxy" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743151 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a6ee20-cbc7-49e8-b581-19310616c0ee" containerName="istio-proxy" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743205 4433 cpu_manager.go:414] "containerMap: removing container" podUID="95740794-649d-4a88-9f4b-ada9e9fe7d2e" containerName="maas-api" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743213 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95740794-649d-4a88-9f4b-ada9e9fe7d2e" containerName="maas-api" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743225 4433 cpu_manager.go:414] "containerMap: removing container" podUID="3ac90de6-4478-4ba6-b106-d40c7c6be7b9" containerName="maas-api" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743234 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac90de6-4478-4ba6-b106-d40c7c6be7b9" containerName="maas-api" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743262 4433 cpu_manager.go:414] "containerMap: removing container" podUID="31b8ebe9-1b26-4870-b9ad-6535916b56f7" containerName="payload-processing" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743269 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b8ebe9-1b26-4870-b9ad-6535916b56f7" containerName="payload-processing" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743313 4433 cpu_manager.go:414] "containerMap: removing container" podUID="d340065c-df73-4f25-bcf9-2b984bdfd710" containerName="maas-api" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743321 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d340065c-df73-4f25-bcf9-2b984bdfd710" containerName="maas-api" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743535 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a48e18-31cf-4e6b-acdb-4f4d9d93d9f3" containerName="istio-proxy" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743553 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2583f5a9-8bc2-45f0-9470-418c96a00d14" containerName="istio-proxy" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743564 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a6ee20-cbc7-49e8-b581-19310616c0ee" containerName="istio-proxy" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743581 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="95740794-649d-4a88-9f4b-ada9e9fe7d2e" containerName="maas-api" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743591 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac90de6-4478-4ba6-b106-d40c7c6be7b9" containerName="maas-api" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743605 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b8ebe9-1b26-4870-b9ad-6535916b56f7" containerName="payload-processing" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743620 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d340065c-df73-4f25-bcf9-2b984bdfd710" containerName="maas-api" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743630 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a62b512-6b1a-4974-8512-eb702c044ecd" containerName="istio-proxy" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743641 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="009b4710-bd3c-4273-bbee-84ebfbe13896" containerName="payload-pre-processing" Oct 30 19:53:15.743900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.743651 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e60493-90ad-4718-a5a5-9acb15f7af19" containerName="istio-proxy" Oct 30 19:53:15.749241 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.749210 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:15.819690 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.819203 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd"] Oct 30 19:53:15.983003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.982745 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b0c827-openshift-default-dockercfg-fxzmx\"" Oct 30 19:53:15.987186 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.986478 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-istio-data\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:15.987186 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.986552 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxxcq\" (UniqueName: \"kubernetes.io/projected/f1e0904a-f62b-4828-b130-fdef755b3831-kube-api-access-vxxcq\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:15.987186 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.986586 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-workload-certs\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:15.987186 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.986617 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e0904a-f62b-4828-b130-fdef755b3831-istio-podinfo\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:15.987186 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.986659 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-credential-socket\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:15.987186 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.986724 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-istio-envoy\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:15.987186 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.986795 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e0904a-f62b-4828-b130-fdef755b3831-istiod-ca-cert\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:15.987186 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.986828 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-workload-socket\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:15.987186 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.986931 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1e0904a-f62b-4828-b130-fdef755b3831-istio-token\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:15.988397 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:15.988372 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:16.087712 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.087679 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-istio-data\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.087880 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.087734 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxxcq\" (UniqueName: \"kubernetes.io/projected/f1e0904a-f62b-4828-b130-fdef755b3831-kube-api-access-vxxcq\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.087880 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.087761 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-workload-certs\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.087880 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.087783 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e0904a-f62b-4828-b130-fdef755b3831-istio-podinfo\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.087880 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.087819 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-credential-socket\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.088155 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.087881 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-istio-envoy\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.088155 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.087956 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e0904a-f62b-4828-b130-fdef755b3831-istiod-ca-cert\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.088155 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.088007 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-workload-socket\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.088155 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.088122 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1e0904a-f62b-4828-b130-fdef755b3831-istio-token\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.088352 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.088183 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-istio-data\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.088352 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.088255 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-workload-certs\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.088542 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.088518 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-workload-socket\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.088663 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.088641 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-credential-socket\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.088784 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.088766 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e0904a-f62b-4828-b130-fdef755b3831-istiod-ca-cert\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.090231 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.090205 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-istio-envoy\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.090324 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.090277 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e0904a-f62b-4828-b130-fdef755b3831-istio-podinfo\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.144550 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.144501 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1e0904a-f62b-4828-b130-fdef755b3831-istio-token\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.144699 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.144678 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxxcq\" (UniqueName: \"kubernetes.io/projected/f1e0904a-f62b-4828-b130-fdef755b3831-kube-api-access-vxxcq\") pod \"e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.396594 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.396500 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:16.397331 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.397296 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:16.830724 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.830695 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd"] Oct 30 19:53:16.832867 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:53:16.832840 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e0904a_f62b_4828_b130_fdef755b3831.slice/crio-8bc8601576984873e5b22667d78f94e30b4487b02d94c56708d0efc1e5ab722a WatchSource:0}: Error finding container 8bc8601576984873e5b22667d78f94e30b4487b02d94c56708d0efc1e5ab722a: Status 404 returned error can't find the container with id 8bc8601576984873e5b22667d78f94e30b4487b02d94c56708d0efc1e5ab722a Oct 30 19:53:16.835575 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.835543 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:53:16.835655 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.835608 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:53:16.835655 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:16.835637 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:53:17.278999 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:17.278903 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:17.609950 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:17.609915 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" event={"ID":"f1e0904a-f62b-4828-b130-fdef755b3831","Type":"ContainerStarted","Data":"035ca861208051e75bb966b5b5cf706a57efee09bd6075ffa690bf847e6f8330"} Oct 30 19:53:17.610191 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:17.609969 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" event={"ID":"f1e0904a-f62b-4828-b130-fdef755b3831","Type":"ContainerStarted","Data":"8bc8601576984873e5b22667d78f94e30b4487b02d94c56708d0efc1e5ab722a"} Oct 30 19:53:17.852056 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:17.839696 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" podStartSLOduration=2.839675726 podStartE2EDuration="2.839675726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:17.836349011 +0000 UTC m=+2244.771136536" watchObservedRunningTime="2026-10-30 19:53:17.839675726 +0000 UTC m=+2244.774463249" Oct 30 19:53:17.901306 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:17.900889 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:18.300671 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:18.300577 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:18.397679 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:18.397647 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:18.403661 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:18.403634 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:18.615789 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:18.615760 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:18.617612 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:18.617590 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:53:19.530746 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:19.530707 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:21.718146 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:21.718102 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:22.002960 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:22.002858 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:22.318733 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:22.318703 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:22.574924 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:22.574838 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:22.878174 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:22.878084 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:23.423848 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:53:23.423809 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:53:23.746529 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:23.746431 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:24.068838 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:24.068807 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:24.431554 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:24.431458 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:26.007816 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:26.007780 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:26.254209 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:26.254122 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:29.153648 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:29.153593 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:30.664781 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:30.664745 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:31.033020 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:31.032866 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:32.985359 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:32.985323 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:33.636553 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:33.636424 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:34.749195 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:34.749164 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:35.043578 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:35.043476 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:35.957828 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:35.957793 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:36.228045 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:36.227948 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:39.946361 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:39.946299 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:40.176842 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:40.176806 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:40.409179 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:40.409133 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:40.640491 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:40.640459 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:43.325145 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:43.325098 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:45.035716 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:45.035682 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:50.375207 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:50.375166 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:53.367122 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:53:53.367085 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:53:53.783072 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:53.783040 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:54.424806 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.424774 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx"] Oct 30 19:53:54.430186 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.430165 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.482164 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.482134 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-75b38d-openshift-default-dockercfg-8b9bw\"" Oct 30 19:53:54.536399 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.536365 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx"] Oct 30 19:53:54.551019 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.550949 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/096a248d-1aed-493d-9ec1-99bf2a8217eb-istiod-ca-cert\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.551198 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.551099 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-workload-certs\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.551198 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.551172 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-data\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.551321 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.551217 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-token\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.551378 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.551251 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-workload-socket\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.551556 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.551533 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrxtk\" (UniqueName: \"kubernetes.io/projected/096a248d-1aed-493d-9ec1-99bf2a8217eb-kube-api-access-lrxtk\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.551673 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.551594 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-credential-socket\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.551759 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.551740 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-envoy\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.551817 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.551777 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-podinfo\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.651045 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.646134 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:54.662623 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.653954 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-data\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.662623 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.654049 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-token\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.662623 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.654092 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-workload-socket\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.662623 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.654189 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrxtk\" (UniqueName: \"kubernetes.io/projected/096a248d-1aed-493d-9ec1-99bf2a8217eb-kube-api-access-lrxtk\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.662623 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.654234 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-credential-socket\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.662623 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.654350 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-envoy\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.662623 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.654385 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-podinfo\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.662623 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.654520 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/096a248d-1aed-493d-9ec1-99bf2a8217eb-istiod-ca-cert\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.662623 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.654595 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-workload-certs\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.662623 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.655817 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-workload-certs\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.662623 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.658362 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-credential-socket\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.662623 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.659762 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-workload-socket\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.669912 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.669525 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-data\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.674106 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.674070 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/096a248d-1aed-493d-9ec1-99bf2a8217eb-istiod-ca-cert\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.688168 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.687940 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-podinfo\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.690749 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.690721 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-envoy\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.718320 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.718288 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-token\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.720217 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.720194 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrxtk\" (UniqueName: \"kubernetes.io/projected/096a248d-1aed-493d-9ec1-99bf2a8217eb-kube-api-access-lrxtk\") pod \"e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.745677 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.745642 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:54.942710 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:54.942608 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:55.180362 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:55.180318 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx"] Oct 30 19:53:55.217563 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:53:55.217531 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096a248d_1aed_493d_9ec1_99bf2a8217eb.slice/crio-e91876d37f81a0cbc28d7174d96c034846faf3357766897f35bf97b13b2b9d9c WatchSource:0}: Error finding container e91876d37f81a0cbc28d7174d96c034846faf3357766897f35bf97b13b2b9d9c: Status 404 returned error can't find the container with id e91876d37f81a0cbc28d7174d96c034846faf3357766897f35bf97b13b2b9d9c Oct 30 19:53:55.220362 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:55.220325 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:53:55.220469 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:55.220413 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:53:55.220469 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:55.220458 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:53:55.347642 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:55.347604 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:55.965439 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:55.965401 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" event={"ID":"096a248d-1aed-493d-9ec1-99bf2a8217eb","Type":"ContainerStarted","Data":"6e66a979d6dd47eaf68da344e7f2c4e5c4cf5f5629edbe513d4a3451ecc01ed9"} Oct 30 19:53:55.965942 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:55.965454 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" event={"ID":"096a248d-1aed-493d-9ec1-99bf2a8217eb","Type":"ContainerStarted","Data":"e91876d37f81a0cbc28d7174d96c034846faf3357766897f35bf97b13b2b9d9c"} Oct 30 19:53:56.124014 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:56.123909 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" podStartSLOduration=2.123891763 podStartE2EDuration="2.123891763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:56.122316225 +0000 UTC m=+2283.057103748" watchObservedRunningTime="2026-10-30 19:53:56.123891763 +0000 UTC m=+2283.058679285" Oct 30 19:53:56.532779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:56.532744 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz"] Oct 30 19:53:56.533032 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:56.533004 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" podUID="4ca58922-83fb-40bb-a289-1fcd9114ffd5" containerName="istio-proxy" containerID="cri-o://388429f376318787e07d067cef219db9385b1ae7a93259ad54629263c9b254e7" gracePeriod=30 Oct 30 19:53:56.746831 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:56.746791 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:56.753394 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:56.753361 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:56.973299 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:56.973270 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:56.974192 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:56.974172 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:53:59.194532 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:59.194488 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:53:59.196909 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:53:59.194544 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" podUID="4ca58922-83fb-40bb-a289-1fcd9114ffd5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:54:00.070921 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:00.070867 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:01.810497 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.810418 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:54:01.967857 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.967822 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-workload-certs\") pod \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " Oct 30 19:54:01.968052 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.967916 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-podinfo\") pod \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " Oct 30 19:54:01.968052 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.967975 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-envoy\") pod \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " Oct 30 19:54:01.968147 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.968121 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-workload-socket\") pod \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " Oct 30 19:54:01.968212 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.968194 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-token\" (UniqueName: \"kubernetes.io/projected/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-token\") pod \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " Oct 30 19:54:01.968267 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.968194 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-workload-certs" pod "4ca58922-83fb-40bb-a289-1fcd9114ffd5" (UID: "4ca58922-83fb-40bb-a289-1fcd9114ffd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:01.968267 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.968259 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-data\") pod \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " Oct 30 19:54:01.968377 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.968306 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-credential-socket\") pod \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " Oct 30 19:54:01.968377 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.968357 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-workload-socket" pod "4ca58922-83fb-40bb-a289-1fcd9114ffd5" (UID: "4ca58922-83fb-40bb-a289-1fcd9114ffd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:01.968480 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.968403 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istiod-ca-cert\") pod \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " Oct 30 19:54:01.968480 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.968450 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca58922-83fb-40bb-a289-1fcd9114ffd5-kube-api-access-lv6sj\" (UniqueName: \"kubernetes.io/projected/4ca58922-83fb-40bb-a289-1fcd9114ffd5-kube-api-access-lv6sj\") pod \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\" (UID: \"4ca58922-83fb-40bb-a289-1fcd9114ffd5\") " Oct 30 19:54:01.968586 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.968482 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-data" pod "4ca58922-83fb-40bb-a289-1fcd9114ffd5" (UID: "4ca58922-83fb-40bb-a289-1fcd9114ffd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:01.968697 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.968648 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-credential-socket" pod "4ca58922-83fb-40bb-a289-1fcd9114ffd5" (UID: "4ca58922-83fb-40bb-a289-1fcd9114ffd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:01.968801 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.968780 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istiod-ca-cert" pod "4ca58922-83fb-40bb-a289-1fcd9114ffd5" (UID: "4ca58922-83fb-40bb-a289-1fcd9114ffd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:01.969244 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.969223 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:54:01.969340 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.969246 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:54:01.969340 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.969260 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:54:01.969340 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.969272 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:54:01.969340 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.969284 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:54:01.970409 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.970382 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-podinfo" pod "4ca58922-83fb-40bb-a289-1fcd9114ffd5" (UID: "4ca58922-83fb-40bb-a289-1fcd9114ffd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:54:01.970511 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.970497 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca58922-83fb-40bb-a289-1fcd9114ffd5-kube-api-access-lv6sj" pod "4ca58922-83fb-40bb-a289-1fcd9114ffd5" (UID: "4ca58922-83fb-40bb-a289-1fcd9114ffd5"). InnerVolumeSpecName "kube-api-access-lv6sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:01.970626 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.970611 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-envoy" pod "4ca58922-83fb-40bb-a289-1fcd9114ffd5" (UID: "4ca58922-83fb-40bb-a289-1fcd9114ffd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:01.970943 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:01.970926 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-token" pod "4ca58922-83fb-40bb-a289-1fcd9114ffd5" (UID: "4ca58922-83fb-40bb-a289-1fcd9114ffd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:02.013878 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:02.013845 4433 generic.go:356] "Generic (PLEG): container finished" podID="4ca58922-83fb-40bb-a289-1fcd9114ffd5" containerID="388429f376318787e07d067cef219db9385b1ae7a93259ad54629263c9b254e7" exitCode=0 Oct 30 19:54:02.014167 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:02.013904 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" Oct 30 19:54:02.014167 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:02.013944 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" event={"ID":"4ca58922-83fb-40bb-a289-1fcd9114ffd5","Type":"ContainerDied","Data":"388429f376318787e07d067cef219db9385b1ae7a93259ad54629263c9b254e7"} Oct 30 19:54:02.014167 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:02.014099 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz" event={"ID":"4ca58922-83fb-40bb-a289-1fcd9114ffd5","Type":"ContainerDied","Data":"727eef303c1dcc60d07d75f546821b3267d5dc81c3b23f4d248ba166a83a9db6"} Oct 30 19:54:02.014167 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:02.014142 4433 scope.go:122] "RemoveContainer" containerID="388429f376318787e07d067cef219db9385b1ae7a93259ad54629263c9b254e7" Oct 30 19:54:02.026057 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:02.026038 4433 scope.go:122] "RemoveContainer" containerID="388429f376318787e07d067cef219db9385b1ae7a93259ad54629263c9b254e7" Oct 30 19:54:02.026365 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:54:02.026341 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388429f376318787e07d067cef219db9385b1ae7a93259ad54629263c9b254e7\": container with ID starting with 388429f376318787e07d067cef219db9385b1ae7a93259ad54629263c9b254e7 not found: ID does not exist" containerID="388429f376318787e07d067cef219db9385b1ae7a93259ad54629263c9b254e7" Oct 30 19:54:02.026412 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:02.026373 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388429f376318787e07d067cef219db9385b1ae7a93259ad54629263c9b254e7"} err="failed to get container status \"388429f376318787e07d067cef219db9385b1ae7a93259ad54629263c9b254e7\": rpc error: code = NotFound desc = could not find container \"388429f376318787e07d067cef219db9385b1ae7a93259ad54629263c9b254e7\": container with ID starting with 388429f376318787e07d067cef219db9385b1ae7a93259ad54629263c9b254e7 not found: ID does not exist" Oct 30 19:54:02.071318 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:02.071282 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:54:02.071318 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:02.071320 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:54:02.071535 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:02.071334 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ca58922-83fb-40bb-a289-1fcd9114ffd5-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:54:02.071535 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:02.071352 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv6sj\" (UniqueName: \"kubernetes.io/projected/4ca58922-83fb-40bb-a289-1fcd9114ffd5-kube-api-access-lv6sj\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:54:02.182155 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:02.182118 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz"] Oct 30 19:54:02.189267 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:02.183906 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-fj6jz"] Oct 30 19:54:03.215407 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:03.215375 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca58922-83fb-40bb-a289-1fcd9114ffd5" path="/var/lib/kubelet/pods/4ca58922-83fb-40bb-a289-1fcd9114ffd5/volumes" Oct 30 19:54:04.724489 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:04.724456 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:08.929381 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:08.929328 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:09.241958 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:09.241836 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:09.622939 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:09.622885 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:12.664246 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:12.664213 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:12.901497 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:12.901459 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:13.138865 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:13.138835 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:13.298103 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:13.298064 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6"] Oct 30 19:54:13.299401 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:13.299344 4433 cpu_manager.go:414] "containerMap: removing container" podUID="4ca58922-83fb-40bb-a289-1fcd9114ffd5" containerName="istio-proxy" Oct 30 19:54:13.301876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:13.301855 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca58922-83fb-40bb-a289-1fcd9114ffd5" containerName="istio-proxy" Oct 30 19:54:13.304068 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:13.304045 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca58922-83fb-40bb-a289-1fcd9114ffd5" containerName="istio-proxy" Oct 30 19:54:13.329819 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:13.329788 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" Oct 30 19:54:13.379852 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:13.379813 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6"] Oct 30 19:54:13.418141 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:13.418061 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-75b38d\"" Oct 30 19:54:13.418490 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:13.418470 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-75b38d-dockercfg-6877w\"" Oct 30 19:54:13.442718 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:13.442081 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whw9b\" (UniqueName: \"kubernetes.io/projected/82dbd69a-c99e-4ea5-b2d3-471707b0d89c-kube-api-access-whw9b\") pod \"payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6\" (UID: \"82dbd69a-c99e-4ea5-b2d3-471707b0d89c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" Oct 30 19:54:13.443178 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:13.443056 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82dbd69a-c99e-4ea5-b2d3-471707b0d89c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6\" (UID: \"82dbd69a-c99e-4ea5-b2d3-471707b0d89c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" Oct 30 19:54:13.545155 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:13.545116 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whw9b\" (UniqueName: \"kubernetes.io/projected/82dbd69a-c99e-4ea5-b2d3-471707b0d89c-kube-api-access-whw9b\") pod \"payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6\" (UID: \"82dbd69a-c99e-4ea5-b2d3-471707b0d89c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" Oct 30 19:54:13.545322 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:13.545237 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82dbd69a-c99e-4ea5-b2d3-471707b0d89c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6\" (UID: \"82dbd69a-c99e-4ea5-b2d3-471707b0d89c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" Oct 30 19:54:13.545922 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:13.545895 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82dbd69a-c99e-4ea5-b2d3-471707b0d89c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6\" (UID: \"82dbd69a-c99e-4ea5-b2d3-471707b0d89c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" Oct 30 19:54:13.601954 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:13.601928 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whw9b\" (UniqueName: \"kubernetes.io/projected/82dbd69a-c99e-4ea5-b2d3-471707b0d89c-kube-api-access-whw9b\") pod \"payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6\" (UID: \"82dbd69a-c99e-4ea5-b2d3-471707b0d89c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" Oct 30 19:54:13.679499 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:13.679427 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" Oct 30 19:54:14.057919 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:14.057890 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6"] Oct 30 19:54:14.058648 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:54:14.058620 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82dbd69a_c99e_4ea5_b2d3_471707b0d89c.slice/crio-5ff7099872861ae760c53ec709dbd2f0cd68c8228ac09f93bf2b7a89b4bb6a4e WatchSource:0}: Error finding container 5ff7099872861ae760c53ec709dbd2f0cd68c8228ac09f93bf2b7a89b4bb6a4e: Status 404 returned error can't find the container with id 5ff7099872861ae760c53ec709dbd2f0cd68c8228ac09f93bf2b7a89b4bb6a4e Oct 30 19:54:14.118355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:14.118320 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" event={"ID":"82dbd69a-c99e-4ea5-b2d3-471707b0d89c","Type":"ContainerStarted","Data":"5ff7099872861ae760c53ec709dbd2f0cd68c8228ac09f93bf2b7a89b4bb6a4e"} Oct 30 19:54:14.467395 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:14.467350 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:15.130085 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:15.130044 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" event={"ID":"82dbd69a-c99e-4ea5-b2d3-471707b0d89c","Type":"ContainerStarted","Data":"289a836b2bf8eb9ce170d040b6af42149e0b3d861fb54b632fe00b7e53453402"} Oct 30 19:54:15.130606 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:15.130125 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" Oct 30 19:54:15.247422 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:15.247369 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" podStartSLOduration=2.247355346 podStartE2EDuration="2.247355346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:15.243840733 +0000 UTC m=+2302.178628257" watchObservedRunningTime="2026-10-30 19:54:15.247355346 +0000 UTC m=+2302.182142870" Oct 30 19:54:19.300203 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:19.300170 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:19.516085 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:19.516034 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:19.733371 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:19.733333 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:23.367004 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:54:23.366940 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:54:24.035119 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:24.035081 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:24.268787 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:24.268755 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:26.142504 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:26.142476 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" Oct 30 19:54:28.413357 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.413323 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g"] Oct 30 19:54:28.417680 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.417659 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:28.471728 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.471697 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-898968-serving-cert\"" Oct 30 19:54:28.472267 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.472196 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-openshift-service-ca\") pod \"maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:28.473652 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.472619 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk82r\" (UniqueName: \"kubernetes.io/projected/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-kube-api-access-lk82r\") pod \"maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:28.473652 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.472712 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-maas-api-tls\") pod \"maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:28.473652 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.472749 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-metrics-certs\") pod \"maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:28.524876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.524843 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g"] Oct 30 19:54:28.574374 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.574333 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk82r\" (UniqueName: \"kubernetes.io/projected/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-kube-api-access-lk82r\") pod \"maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:28.574573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.574416 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-maas-api-tls\") pod \"maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:28.574573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.574466 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-metrics-certs\") pod \"maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:28.574573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.574519 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-openshift-service-ca\") pod \"maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:28.574573 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:54:28.574555 4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-serving-cert: secret "maas-api-e2e-inf-a-898968-serving-cert" not found Oct 30 19:54:28.574767 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:54:28.574613 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-maas-api-tls podName:7a6f7dc5-3d1d-4b4b-b714-cded507fb414 nodeName:}" failed. No retries permitted until 2026-10-30 19:54:29.074598787 +0000 UTC m=+2316.009386294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-maas-api-tls") pod "maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" (UID: "7a6f7dc5-3d1d-4b4b-b714-cded507fb414") : secret "maas-api-e2e-inf-a-898968-serving-cert" not found Oct 30 19:54:28.575308 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.575276 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-openshift-service-ca\") pod \"maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:28.577171 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.577152 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-metrics-certs\") pod \"maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:28.634193 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.634162 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk82r\" (UniqueName: \"kubernetes.io/projected/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-kube-api-access-lk82r\") pod \"maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:28.690602 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.690505 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:28.908371 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:28.908337 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:29.081786 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:29.081741 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-maas-api-tls\") pod \"maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:29.084372 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:29.084348 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-maas-api-tls\") pod \"maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:29.331622 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:29.331581 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:29.708258 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:29.708223 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g"] Oct 30 19:54:29.709389 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:54:29.709363 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6f7dc5_3d1d_4b4b_b714_cded507fb414.slice/crio-8570c8e8b360368e0707b4ef1638a1488b2015aa66a7a01fddc330aa3e0bd44f WatchSource:0}: Error finding container 8570c8e8b360368e0707b4ef1638a1488b2015aa66a7a01fddc330aa3e0bd44f: Status 404 returned error can't find the container with id 8570c8e8b360368e0707b4ef1638a1488b2015aa66a7a01fddc330aa3e0bd44f Oct 30 19:54:29.710808 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:29.710791 4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:54:30.234271 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:30.234237 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" event={"ID":"7a6f7dc5-3d1d-4b4b-b714-cded507fb414","Type":"ContainerStarted","Data":"8570c8e8b360368e0707b4ef1638a1488b2015aa66a7a01fddc330aa3e0bd44f"} Oct 30 19:54:31.242354 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:31.242313 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" event={"ID":"7a6f7dc5-3d1d-4b4b-b714-cded507fb414","Type":"ContainerStarted","Data":"1cdeaa658539870fc608231de49cb87a5aecc8b02cf9cb5106cff8d0e0ae84c5"} Oct 30 19:54:31.242762 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:31.242404 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:31.360344 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:31.360284 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" podStartSLOduration=2.525151752 podStartE2EDuration="3.360267512s" totalImagesPullingTime="835.11576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:31.355577413 +0000 UTC m=+2318.290364938" watchObservedRunningTime="2026-10-30 19:54:31.360267512 +0000 UTC m=+2318.295055036" Oct 30 19:54:31.826001 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:31.825868 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:32.048454 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:32.048420 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:33.771773 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:33.771731 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:33.993278 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:33.993249 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:37.258604 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:37.258575 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:54:38.360873 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:38.360841 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:42.985272 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:42.985238 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:46.478949 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:46.478907 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:47.745805 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:47.745765 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:51.973371 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:51.973339 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv"] Oct 30 19:54:51.978283 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:51.978261 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:54:52.080973 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.080940 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-da4c62-serving-cert\"" Oct 30 19:54:52.084600 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.084575 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv"] Oct 30 19:54:52.092118 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.092092 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-maas-api-tls\") pod \"maas-api-e2e-api-b-da4c62-5499dd465-ld8tv\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:54:52.092225 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.092138 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqbqv\" (UniqueName: \"kubernetes.io/projected/e1409587-03bd-41d3-93cb-83d38bd9a4f2-kube-api-access-xqbqv\") pod \"maas-api-e2e-api-b-da4c62-5499dd465-ld8tv\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:54:52.092225 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.092202 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1409587-03bd-41d3-93cb-83d38bd9a4f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-da4c62-5499dd465-ld8tv\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:54:52.092382 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.092363 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-metrics-certs\") pod \"maas-api-e2e-api-b-da4c62-5499dd465-ld8tv\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:54:52.193881 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.193852 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-maas-api-tls\") pod \"maas-api-e2e-api-b-da4c62-5499dd465-ld8tv\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:54:52.194044 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.193905 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqbqv\" (UniqueName: \"kubernetes.io/projected/e1409587-03bd-41d3-93cb-83d38bd9a4f2-kube-api-access-xqbqv\") pod \"maas-api-e2e-api-b-da4c62-5499dd465-ld8tv\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:54:52.194044 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.193944 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1409587-03bd-41d3-93cb-83d38bd9a4f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-da4c62-5499dd465-ld8tv\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:54:52.194044 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:54:52.194007 4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-serving-cert: secret "maas-api-e2e-api-b-da4c62-serving-cert" not found Oct 30 19:54:52.194044 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.194026 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-metrics-certs\") pod \"maas-api-e2e-api-b-da4c62-5499dd465-ld8tv\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:54:52.194234 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:54:52.194081 4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-maas-api-tls podName:e1409587-03bd-41d3-93cb-83d38bd9a4f2 nodeName:}" failed. No retries permitted until 2026-10-30 19:54:52.694063317 +0000 UTC m=+2339.628850821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-maas-api-tls") pod "maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" (UID: "e1409587-03bd-41d3-93cb-83d38bd9a4f2") : secret "maas-api-e2e-api-b-da4c62-serving-cert" not found Oct 30 19:54:52.194790 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.194761 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1409587-03bd-41d3-93cb-83d38bd9a4f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-da4c62-5499dd465-ld8tv\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:54:52.196502 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.196477 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-metrics-certs\") pod \"maas-api-e2e-api-b-da4c62-5499dd465-ld8tv\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:54:52.255424 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.255338 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqbqv\" (UniqueName: \"kubernetes.io/projected/e1409587-03bd-41d3-93cb-83d38bd9a4f2-kube-api-access-xqbqv\") pod \"maas-api-e2e-api-b-da4c62-5499dd465-ld8tv\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:54:52.259094 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.257868 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:52.491151 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.491106 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:52.700454 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.700415 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-maas-api-tls\") pod \"maas-api-e2e-api-b-da4c62-5499dd465-ld8tv\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:54:52.702858 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.702827 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-maas-api-tls\") pod \"maas-api-e2e-api-b-da4c62-5499dd465-ld8tv\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:54:52.706444 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.706416 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:52.892188 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:52.892167 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:54:53.279824 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:53.279768 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv"] Oct 30 19:54:53.424061 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:53.424014 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" event={"ID":"e1409587-03bd-41d3-93cb-83d38bd9a4f2","Type":"ContainerStarted","Data":"30a8f8b765a48e5b6efac2122efef396edd8903763d63b19a4542a011c1b8b1e"} Oct 30 19:54:53.469193 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:54:53.469150 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:54:54.432475 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:54.432364 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" event={"ID":"e1409587-03bd-41d3-93cb-83d38bd9a4f2","Type":"ContainerStarted","Data":"6ffc52b3706d44e2f6e809c62ffc803fb440175bbfb3033ba76aa1c0fde4c186"} Oct 30 19:54:54.432475 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:54.432434 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:54:54.571187 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:54.571120 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" podStartSLOduration=2.721667428 podStartE2EDuration="3.571097071s" totalImagesPullingTime="849.429643ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:54.554416788 +0000 UTC m=+2341.489204338" watchObservedRunningTime="2026-10-30 19:54:54.571097071 +0000 UTC m=+2341.505884597" Oct 30 19:54:57.156854 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:54:57.156820 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:00.460368 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:00.460333 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:55:01.938611 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:01.938577 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:02.185211 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:02.185176 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:02.402581 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:02.402521 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:06.832814 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:06.832779 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:07.774176 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:07.774137 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm"] Oct 30 19:55:07.774369 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:07.774338 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" podUID="abe0d869-8b60-4ef3-acf4-8a0c2375b35c" containerName="istio-proxy" containerID="cri-o://d7d53f608a0c73cd0cc76fef46666a61c24cc30e2a9593db5e6cb6e5da7e8008" gracePeriod=30 Oct 30 19:55:08.453813 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:08.453759 4433 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:55:08.454230 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:08.453849 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" podUID="abe0d869-8b60-4ef3-acf4-8a0c2375b35c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:55:11.434932 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:11.434893 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:11.664459 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:11.664416 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:13.042338 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.042282 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:55:13.127106 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.127078 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-data\") pod \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " Oct 30 19:55:13.127230 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.127118 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-credential-socket\") pod \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " Oct 30 19:55:13.127230 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.127147 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-envoy\") pod \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " Oct 30 19:55:13.127230 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.127185 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istiod-ca-cert\") pod \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " Oct 30 19:55:13.127357 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.127242 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-kube-api-access-sb7ln\" (UniqueName: \"kubernetes.io/projected/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-kube-api-access-sb7ln\") pod \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " Oct 30 19:55:13.127357 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.127270 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-workload-socket\") pod \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " Oct 30 19:55:13.127357 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.127326 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-token\" (UniqueName: \"kubernetes.io/projected/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-token\") pod \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " Oct 30 19:55:13.127504 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.127378 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-workload-certs\") pod \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " Oct 30 19:55:13.127504 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.127385 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-data" pod "abe0d869-8b60-4ef3-acf4-8a0c2375b35c" (UID: "abe0d869-8b60-4ef3-acf4-8a0c2375b35c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:13.127504 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.127409 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-podinfo\") pod \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\" (UID: \"abe0d869-8b60-4ef3-acf4-8a0c2375b35c\") " Oct 30 19:55:13.127504 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.127409 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-credential-socket" pod "abe0d869-8b60-4ef3-acf4-8a0c2375b35c" (UID: "abe0d869-8b60-4ef3-acf4-8a0c2375b35c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:13.127712 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.127672 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istiod-ca-cert" pod "abe0d869-8b60-4ef3-acf4-8a0c2375b35c" (UID: "abe0d869-8b60-4ef3-acf4-8a0c2375b35c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:13.127712 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.127661 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-workload-certs" pod "abe0d869-8b60-4ef3-acf4-8a0c2375b35c" (UID: "abe0d869-8b60-4ef3-acf4-8a0c2375b35c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:13.127712 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.127698 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-workload-socket" pod "abe0d869-8b60-4ef3-acf4-8a0c2375b35c" (UID: "abe0d869-8b60-4ef3-acf4-8a0c2375b35c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:13.128191 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.128166 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.128311 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.128195 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.128311 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.128211 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.128311 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.128250 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.128311 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.128265 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.129499 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.129474 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-token" pod "abe0d869-8b60-4ef3-acf4-8a0c2375b35c" (UID: "abe0d869-8b60-4ef3-acf4-8a0c2375b35c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:13.129582 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.129489 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-envoy" pod "abe0d869-8b60-4ef3-acf4-8a0c2375b35c" (UID: "abe0d869-8b60-4ef3-acf4-8a0c2375b35c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:13.129635 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.129607 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-kube-api-access-sb7ln" pod "abe0d869-8b60-4ef3-acf4-8a0c2375b35c" (UID: "abe0d869-8b60-4ef3-acf4-8a0c2375b35c"). InnerVolumeSpecName "kube-api-access-sb7ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:13.129672 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.129659 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-podinfo" pod "abe0d869-8b60-4ef3-acf4-8a0c2375b35c" (UID: "abe0d869-8b60-4ef3-acf4-8a0c2375b35c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:55:13.237457 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.236965 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.237457 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.237023 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb7ln\" (UniqueName: \"kubernetes.io/projected/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-kube-api-access-sb7ln\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.237457 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.237038 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.237457 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.237056 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe0d869-8b60-4ef3-acf4-8a0c2375b35c-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.619355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.619317 4433 generic.go:356] "Generic (PLEG): container finished" podID="abe0d869-8b60-4ef3-acf4-8a0c2375b35c" containerID="d7d53f608a0c73cd0cc76fef46666a61c24cc30e2a9593db5e6cb6e5da7e8008" exitCode=0 Oct 30 19:55:13.619355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.619347 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" event={"ID":"abe0d869-8b60-4ef3-acf4-8a0c2375b35c","Type":"ContainerDied","Data":"d7d53f608a0c73cd0cc76fef46666a61c24cc30e2a9593db5e6cb6e5da7e8008"} Oct 30 19:55:13.619576 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.619369 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" Oct 30 19:55:13.619576 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.619402 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm" event={"ID":"abe0d869-8b60-4ef3-acf4-8a0c2375b35c","Type":"ContainerDied","Data":"06e6623acfed759fd5eba22bf5e31d175d568452797fd5f4f66ac414db8bfac4"} Oct 30 19:55:13.619576 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.619424 4433 scope.go:122] "RemoveContainer" containerID="d7d53f608a0c73cd0cc76fef46666a61c24cc30e2a9593db5e6cb6e5da7e8008" Oct 30 19:55:13.630209 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.630185 4433 scope.go:122] "RemoveContainer" containerID="d7d53f608a0c73cd0cc76fef46666a61c24cc30e2a9593db5e6cb6e5da7e8008" Oct 30 19:55:13.630487 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:55:13.630471 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d53f608a0c73cd0cc76fef46666a61c24cc30e2a9593db5e6cb6e5da7e8008\": container with ID starting with d7d53f608a0c73cd0cc76fef46666a61c24cc30e2a9593db5e6cb6e5da7e8008 not found: ID does not exist" containerID="d7d53f608a0c73cd0cc76fef46666a61c24cc30e2a9593db5e6cb6e5da7e8008" Oct 30 19:55:13.630535 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.630495 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d53f608a0c73cd0cc76fef46666a61c24cc30e2a9593db5e6cb6e5da7e8008"} err="failed to get container status \"d7d53f608a0c73cd0cc76fef46666a61c24cc30e2a9593db5e6cb6e5da7e8008\": rpc error: code = NotFound desc = could not find container \"d7d53f608a0c73cd0cc76fef46666a61c24cc30e2a9593db5e6cb6e5da7e8008\": container with ID starting with d7d53f608a0c73cd0cc76fef46666a61c24cc30e2a9593db5e6cb6e5da7e8008 not found: ID does not exist" Oct 30 19:55:13.787099 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.787056 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm"] Oct 30 19:55:13.836107 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:13.836077 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-kgncm"] Oct 30 19:55:15.215698 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:15.215666 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abe0d869-8b60-4ef3-acf4-8a0c2375b35c" path="/var/lib/kubelet/pods/abe0d869-8b60-4ef3-acf4-8a0c2375b35c/volumes" Oct 30 19:55:16.550520 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:16.550481 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:19.910860 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:19.910824 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:21.189683 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:21.189641 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:23.371499 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:55:23.371461 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:55:25.867102 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:25.867065 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:29.931841 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:29.931799 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:31.543779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:31.543743 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:31.926281 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:31.926249 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:32.148518 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:32.148485 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:34.308272 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:34.308237 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:34.658151 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:34.658104 4433 patch_prober.go:29] interesting pod/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:55:34.658352 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:34.658168 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" podUID="f1e0904a-f62b-4828-b130-fdef755b3831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:35.834775 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:35.834740 4433 generic.go:356] "Generic (PLEG): container finished" podID="f1e0904a-f62b-4828-b130-fdef755b3831" containerID="035ca861208051e75bb966b5b5cf706a57efee09bd6075ffa690bf847e6f8330" exitCode=0 Oct 30 19:55:35.835283 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:35.834809 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" event={"ID":"f1e0904a-f62b-4828-b130-fdef755b3831","Type":"ContainerDied","Data":"035ca861208051e75bb966b5b5cf706a57efee09bd6075ffa690bf847e6f8330"} Oct 30 19:55:35.835387 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:35.835360 4433 scope.go:122] "RemoveContainer" containerID="035ca861208051e75bb966b5b5cf706a57efee09bd6075ffa690bf847e6f8330" Oct 30 19:55:35.837391 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:35.837361 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:55:35.837490 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:35.837434 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:55:35.837490 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:35.837465 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:55:36.069257 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:36.069223 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:36.843731 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:36.843698 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" event={"ID":"f1e0904a-f62b-4828-b130-fdef755b3831","Type":"ContainerStarted","Data":"b7565739b966ce2722d086cd53855837ae23c0b7ee2258b0125eb27961369afb"} Oct 30 19:55:37.397419 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:37.397385 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:55:37.397776 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:37.397752 4433 patch_prober.go:29] interesting pod/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:55:37.397919 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:37.397896 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" podUID="f1e0904a-f62b-4828-b130-fdef755b3831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 19:55:37.649453 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:37.649368 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:37.822889 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:37.822857 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9"] Oct 30 19:55:37.823151 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:37.823066 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" podUID="34bbe99d-caf6-46cb-a4b4-622ee14e8d75" containerName="istio-proxy" containerID="cri-o://6d91a3053fcdd29137f4458870a68b866b73eaa52c0bc7944fe970c73952ec8d" gracePeriod=30 Oct 30 19:55:38.013777 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:38.013693 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s"] Oct 30 19:55:38.014239 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:38.013898 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" podUID="335bb767-624e-44ba-8dab-3bab19297c9e" containerName="istio-proxy" containerID="cri-o://851e8cd36984587096d208da4f9bd59200f65bd45d80bf281c1043c0060327f3" gracePeriod=30 Oct 30 19:55:38.233792 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:38.233755 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:38.405309 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:38.405268 4433 patch_prober.go:29] interesting pod/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:55:38.405610 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:38.405581 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" podUID="f1e0904a-f62b-4828-b130-fdef755b3831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 19:55:39.409094 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:39.409061 4433 patch_prober.go:29] interesting pod/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:55:39.409513 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:39.409490 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" podUID="f1e0904a-f62b-4828-b130-fdef755b3831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 19:55:39.534657 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:39.534629 4433 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:55:39.534876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:39.534856 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" podUID="34bbe99d-caf6-46cb-a4b4-622ee14e8d75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:55:39.709052 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:39.707456 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:40.002975 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:40.002872 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:40.404405 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:40.404364 4433 patch_prober.go:29] interesting pod/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:55:40.404730 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:40.404695 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" podUID="f1e0904a-f62b-4828-b130-fdef755b3831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 19:55:41.183885 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:41.183850 4433 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:55:41.184397 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:41.183905 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" podUID="335bb767-624e-44ba-8dab-3bab19297c9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:55:41.405945 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:41.405906 4433 patch_prober.go:29] interesting pod/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:55:41.411007 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:41.410144 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" podUID="f1e0904a-f62b-4828-b130-fdef755b3831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 19:55:41.879131 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:41.879094 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:42.214450 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:42.214352 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:42.431651 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:42.430262 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:55:42.431651 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:42.431481 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:55:42.446304 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:42.446221 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 19:55:42.473061 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:42.468484 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:43.369275 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.369246 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:55:43.372859 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.372785 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:55:43.402830 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.402794 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-istio-data\") pod \"335bb767-624e-44ba-8dab-3bab19297c9e\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " Oct 30 19:55:43.403041 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.402844 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-workload-socket\") pod \"335bb767-624e-44ba-8dab-3bab19297c9e\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " Oct 30 19:55:43.403041 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.402898 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-workload-certs\") pod \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " Oct 30 19:55:43.403041 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.402928 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-credential-socket\") pod \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " Oct 30 19:55:43.403220 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403093 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-istio-data" pod "335bb767-624e-44ba-8dab-3bab19297c9e" (UID: "335bb767-624e-44ba-8dab-3bab19297c9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:43.403220 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403169 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istiod-ca-cert\") pod \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " Oct 30 19:55:43.403220 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403203 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/335bb767-624e-44ba-8dab-3bab19297c9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335bb767-624e-44ba-8dab-3bab19297c9e-istiod-ca-cert\") pod \"335bb767-624e-44ba-8dab-3bab19297c9e\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " Oct 30 19:55:43.403372 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403211 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-workload-socket" pod "335bb767-624e-44ba-8dab-3bab19297c9e" (UID: "335bb767-624e-44ba-8dab-3bab19297c9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:43.403372 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403228 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-credential-socket" pod "34bbe99d-caf6-46cb-a4b4-622ee14e8d75" (UID: "34bbe99d-caf6-46cb-a4b4-622ee14e8d75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:43.403372 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403236 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335bb767-624e-44ba-8dab-3bab19297c9e-kube-api-access-w5snz\" (UniqueName: \"kubernetes.io/projected/335bb767-624e-44ba-8dab-3bab19297c9e-kube-api-access-w5snz\") pod \"335bb767-624e-44ba-8dab-3bab19297c9e\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " Oct 30 19:55:43.403372 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403338 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-workload-certs\") pod \"335bb767-624e-44ba-8dab-3bab19297c9e\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " Oct 30 19:55:43.403568 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403436 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-podinfo\") pod \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " Oct 30 19:55:43.403568 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403504 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-istio-envoy\") pod \"335bb767-624e-44ba-8dab-3bab19297c9e\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " Oct 30 19:55:43.403568 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403546 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-credential-socket\") pod \"335bb767-624e-44ba-8dab-3bab19297c9e\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " Oct 30 19:55:43.403816 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403574 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-token\" (UniqueName: \"kubernetes.io/projected/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-token\") pod \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " Oct 30 19:55:43.403816 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403576 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istiod-ca-cert" pod "34bbe99d-caf6-46cb-a4b4-622ee14e8d75" (UID: "34bbe99d-caf6-46cb-a4b4-622ee14e8d75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:43.403816 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403604 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-data\") pod \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " Oct 30 19:55:43.403816 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403678 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-workload-certs" pod "34bbe99d-caf6-46cb-a4b4-622ee14e8d75" (UID: "34bbe99d-caf6-46cb-a4b4-622ee14e8d75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:43.403816 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403744 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/335bb767-624e-44ba-8dab-3bab19297c9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335bb767-624e-44ba-8dab-3bab19297c9e-istio-podinfo\") pod \"335bb767-624e-44ba-8dab-3bab19297c9e\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " Oct 30 19:55:43.403816 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403784 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-data" pod "34bbe99d-caf6-46cb-a4b4-622ee14e8d75" (UID: "34bbe99d-caf6-46cb-a4b4-622ee14e8d75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:43.403816 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403791 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-kube-api-access-hdkrt\" (UniqueName: \"kubernetes.io/projected/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-kube-api-access-hdkrt\") pod \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " Oct 30 19:55:43.404236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403893 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-envoy\") pod \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " Oct 30 19:55:43.404236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403964 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-workload-socket\") pod \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\" (UID: \"34bbe99d-caf6-46cb-a4b4-622ee14e8d75\") " Oct 30 19:55:43.404236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.403999 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-workload-certs" pod "335bb767-624e-44ba-8dab-3bab19297c9e" (UID: "335bb767-624e-44ba-8dab-3bab19297c9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:43.404236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.404056 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335bb767-624e-44ba-8dab-3bab19297c9e-istio-token\" (UniqueName: \"kubernetes.io/projected/335bb767-624e-44ba-8dab-3bab19297c9e-istio-token\") pod \"335bb767-624e-44ba-8dab-3bab19297c9e\" (UID: \"335bb767-624e-44ba-8dab-3bab19297c9e\") " Oct 30 19:55:43.404236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.404080 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335bb767-624e-44ba-8dab-3bab19297c9e-istiod-ca-cert" pod "335bb767-624e-44ba-8dab-3bab19297c9e" (UID: "335bb767-624e-44ba-8dab-3bab19297c9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:43.404564 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.404520 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-credential-socket" pod "335bb767-624e-44ba-8dab-3bab19297c9e" (UID: "335bb767-624e-44ba-8dab-3bab19297c9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:43.405301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.404876 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.405301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.404897 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.405301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.404912 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.405301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.404927 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.405301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.404942 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.405301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.404955 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335bb767-624e-44ba-8dab-3bab19297c9e-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.405301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.404969 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.405301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.405016 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.405301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.405029 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.406313 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.406082 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335bb767-624e-44ba-8dab-3bab19297c9e-kube-api-access-w5snz" pod "335bb767-624e-44ba-8dab-3bab19297c9e" (UID: "335bb767-624e-44ba-8dab-3bab19297c9e"). InnerVolumeSpecName "kube-api-access-w5snz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:43.406549 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.406467 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-workload-socket" pod "34bbe99d-caf6-46cb-a4b4-622ee14e8d75" (UID: "34bbe99d-caf6-46cb-a4b4-622ee14e8d75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:43.406920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.406887 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-kube-api-access-hdkrt" pod "34bbe99d-caf6-46cb-a4b4-622ee14e8d75" (UID: "34bbe99d-caf6-46cb-a4b4-622ee14e8d75"). InnerVolumeSpecName "kube-api-access-hdkrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:43.408751 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.408438 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335bb767-624e-44ba-8dab-3bab19297c9e-istio-token" pod "335bb767-624e-44ba-8dab-3bab19297c9e" (UID: "335bb767-624e-44ba-8dab-3bab19297c9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:43.409391 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.409266 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/335bb767-624e-44ba-8dab-3bab19297c9e-istio-podinfo" pod "335bb767-624e-44ba-8dab-3bab19297c9e" (UID: "335bb767-624e-44ba-8dab-3bab19297c9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:55:43.409391 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.409308 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-envoy" pod "34bbe99d-caf6-46cb-a4b4-622ee14e8d75" (UID: "34bbe99d-caf6-46cb-a4b4-622ee14e8d75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:43.409391 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.409358 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-istio-envoy" pod "335bb767-624e-44ba-8dab-3bab19297c9e" (UID: "335bb767-624e-44ba-8dab-3bab19297c9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:43.409589 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.409571 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-podinfo" pod "34bbe99d-caf6-46cb-a4b4-622ee14e8d75" (UID: "34bbe99d-caf6-46cb-a4b4-622ee14e8d75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:55:43.410230 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.410207 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-token" pod "34bbe99d-caf6-46cb-a4b4-622ee14e8d75" (UID: "34bbe99d-caf6-46cb-a4b4-622ee14e8d75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:43.506180 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.506092 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335bb767-624e-44ba-8dab-3bab19297c9e-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.506180 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.506121 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.506180 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.506133 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335bb767-624e-44ba-8dab-3bab19297c9e-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.506180 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.506142 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdkrt\" (UniqueName: \"kubernetes.io/projected/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-kube-api-access-hdkrt\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.506180 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.506154 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.506180 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.506162 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.506180 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.506172 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335bb767-624e-44ba-8dab-3bab19297c9e-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.506180 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.506182 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5snz\" (UniqueName: \"kubernetes.io/projected/335bb767-624e-44ba-8dab-3bab19297c9e-kube-api-access-w5snz\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.506573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.506191 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bbe99d-caf6-46cb-a4b4-622ee14e8d75-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.997752 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.997714 4433 generic.go:356] "Generic (PLEG): container finished" podID="34bbe99d-caf6-46cb-a4b4-622ee14e8d75" containerID="6d91a3053fcdd29137f4458870a68b866b73eaa52c0bc7944fe970c73952ec8d" exitCode=0 Oct 30 19:55:43.997916 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.997769 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" Oct 30 19:55:43.997916 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.997803 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" event={"ID":"34bbe99d-caf6-46cb-a4b4-622ee14e8d75","Type":"ContainerDied","Data":"6d91a3053fcdd29137f4458870a68b866b73eaa52c0bc7944fe970c73952ec8d"} Oct 30 19:55:43.997916 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.997861 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9" event={"ID":"34bbe99d-caf6-46cb-a4b4-622ee14e8d75","Type":"ContainerDied","Data":"48f67ed9466bd6006cd619cefb7924d7b1d3d412a3d39652c66202e4451a217b"} Oct 30 19:55:43.997916 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.997891 4433 scope.go:122] "RemoveContainer" containerID="6d91a3053fcdd29137f4458870a68b866b73eaa52c0bc7944fe970c73952ec8d" Oct 30 19:55:43.999432 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.999405 4433 generic.go:356] "Generic (PLEG): container finished" podID="335bb767-624e-44ba-8dab-3bab19297c9e" containerID="851e8cd36984587096d208da4f9bd59200f65bd45d80bf281c1043c0060327f3" exitCode=0 Oct 30 19:55:43.999524 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.999467 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" Oct 30 19:55:43.999524 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.999492 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" event={"ID":"335bb767-624e-44ba-8dab-3bab19297c9e","Type":"ContainerDied","Data":"851e8cd36984587096d208da4f9bd59200f65bd45d80bf281c1043c0060327f3"} Oct 30 19:55:43.999645 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:43.999538 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s" event={"ID":"335bb767-624e-44ba-8dab-3bab19297c9e","Type":"ContainerDied","Data":"fa59063059b1ddc348776f9e476f1ba71ae822230950cc77298c8d30c40c5e92"} Oct 30 19:55:44.010267 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:44.010131 4433 scope.go:122] "RemoveContainer" containerID="6d91a3053fcdd29137f4458870a68b866b73eaa52c0bc7944fe970c73952ec8d" Oct 30 19:55:44.010501 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:55:44.010479 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d91a3053fcdd29137f4458870a68b866b73eaa52c0bc7944fe970c73952ec8d\": container with ID starting with 6d91a3053fcdd29137f4458870a68b866b73eaa52c0bc7944fe970c73952ec8d not found: ID does not exist" containerID="6d91a3053fcdd29137f4458870a68b866b73eaa52c0bc7944fe970c73952ec8d" Oct 30 19:55:44.010571 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:44.010513 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d91a3053fcdd29137f4458870a68b866b73eaa52c0bc7944fe970c73952ec8d"} err="failed to get container status \"6d91a3053fcdd29137f4458870a68b866b73eaa52c0bc7944fe970c73952ec8d\": rpc error: code = NotFound desc = could not find container \"6d91a3053fcdd29137f4458870a68b866b73eaa52c0bc7944fe970c73952ec8d\": container with ID starting with 6d91a3053fcdd29137f4458870a68b866b73eaa52c0bc7944fe970c73952ec8d not found: ID does not exist" Oct 30 19:55:44.010571 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:44.010538 4433 scope.go:122] "RemoveContainer" containerID="851e8cd36984587096d208da4f9bd59200f65bd45d80bf281c1043c0060327f3" Oct 30 19:55:44.024329 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:44.024300 4433 scope.go:122] "RemoveContainer" containerID="851e8cd36984587096d208da4f9bd59200f65bd45d80bf281c1043c0060327f3" Oct 30 19:55:44.024715 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:55:44.024679 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851e8cd36984587096d208da4f9bd59200f65bd45d80bf281c1043c0060327f3\": container with ID starting with 851e8cd36984587096d208da4f9bd59200f65bd45d80bf281c1043c0060327f3 not found: ID does not exist" containerID="851e8cd36984587096d208da4f9bd59200f65bd45d80bf281c1043c0060327f3" Oct 30 19:55:44.024789 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:44.024727 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851e8cd36984587096d208da4f9bd59200f65bd45d80bf281c1043c0060327f3"} err="failed to get container status \"851e8cd36984587096d208da4f9bd59200f65bd45d80bf281c1043c0060327f3\": rpc error: code = NotFound desc = could not find container \"851e8cd36984587096d208da4f9bd59200f65bd45d80bf281c1043c0060327f3\": container with ID starting with 851e8cd36984587096d208da4f9bd59200f65bd45d80bf281c1043c0060327f3 not found: ID does not exist" Oct 30 19:55:44.162406 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:44.162298 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s"] Oct 30 19:55:44.213418 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:44.213373 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-prr9s"] Oct 30 19:55:44.274359 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:44.274326 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9"] Oct 30 19:55:44.277754 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:44.277726 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-k5bl9"] Oct 30 19:55:45.215591 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:45.215560 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="335bb767-624e-44ba-8dab-3bab19297c9e" path="/var/lib/kubelet/pods/335bb767-624e-44ba-8dab-3bab19297c9e/volumes" Oct 30 19:55:45.216150 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:45.216135 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34bbe99d-caf6-46cb-a4b4-622ee14e8d75" path="/var/lib/kubelet/pods/34bbe99d-caf6-46cb-a4b4-622ee14e8d75/volumes" Oct 30 19:55:45.321754 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:45.321714 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5"] Oct 30 19:55:45.321960 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:45.321937 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" podUID="367229e1-f694-4449-9e91-eadaf2502938" containerName="istio-proxy" containerID="cri-o://c034c45611b330f8e24a09b20f826204843342f85b43de54ff79a3f41715b897" gracePeriod=30 Oct 30 19:55:46.782713 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:46.782581 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:49.236967 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:49.236926 4433 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:55:49.237433 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:49.237008 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" podUID="367229e1-f694-4449-9e91-eadaf2502938" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:55:50.605342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.605273 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:55:50.719911 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.719869 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/367229e1-f694-4449-9e91-eadaf2502938-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367229e1-f694-4449-9e91-eadaf2502938-istio-podinfo\") pod \"367229e1-f694-4449-9e91-eadaf2502938\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " Oct 30 19:55:50.720166 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.720058 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367229e1-f694-4449-9e91-eadaf2502938-istio-token\" (UniqueName: \"kubernetes.io/projected/367229e1-f694-4449-9e91-eadaf2502938-istio-token\") pod \"367229e1-f694-4449-9e91-eadaf2502938\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " Oct 30 19:55:50.720166 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.720159 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367229e1-f694-4449-9e91-eadaf2502938-kube-api-access-5fqnt\" (UniqueName: \"kubernetes.io/projected/367229e1-f694-4449-9e91-eadaf2502938-kube-api-access-5fqnt\") pod \"367229e1-f694-4449-9e91-eadaf2502938\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " Oct 30 19:55:50.720288 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.720195 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-workload-certs\") pod \"367229e1-f694-4449-9e91-eadaf2502938\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " Oct 30 19:55:50.720288 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.720244 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-workload-socket\") pod \"367229e1-f694-4449-9e91-eadaf2502938\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " Oct 30 19:55:50.720381 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.720303 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/367229e1-f694-4449-9e91-eadaf2502938-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367229e1-f694-4449-9e91-eadaf2502938-istiod-ca-cert\") pod \"367229e1-f694-4449-9e91-eadaf2502938\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " Oct 30 19:55:50.720487 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.720447 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-istio-data\") pod \"367229e1-f694-4449-9e91-eadaf2502938\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " Oct 30 19:55:50.720540 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.720508 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-credential-socket\") pod \"367229e1-f694-4449-9e91-eadaf2502938\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " Oct 30 19:55:50.720612 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.720594 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-istio-envoy\") pod \"367229e1-f694-4449-9e91-eadaf2502938\" (UID: \"367229e1-f694-4449-9e91-eadaf2502938\") " Oct 30 19:55:50.721221 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.721172 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-credential-socket" pod "367229e1-f694-4449-9e91-eadaf2502938" (UID: "367229e1-f694-4449-9e91-eadaf2502938"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:50.721335 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.721219 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-workload-certs" pod "367229e1-f694-4449-9e91-eadaf2502938" (UID: "367229e1-f694-4449-9e91-eadaf2502938"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:50.721335 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.721239 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-istio-data" pod "367229e1-f694-4449-9e91-eadaf2502938" (UID: "367229e1-f694-4449-9e91-eadaf2502938"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:50.721335 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.721274 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/367229e1-f694-4449-9e91-eadaf2502938-istiod-ca-cert" pod "367229e1-f694-4449-9e91-eadaf2502938" (UID: "367229e1-f694-4449-9e91-eadaf2502938"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:50.721503 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.721394 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-workload-socket" pod "367229e1-f694-4449-9e91-eadaf2502938" (UID: "367229e1-f694-4449-9e91-eadaf2502938"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:50.723363 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.723052 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/367229e1-f694-4449-9e91-eadaf2502938-istio-podinfo" pod "367229e1-f694-4449-9e91-eadaf2502938" (UID: "367229e1-f694-4449-9e91-eadaf2502938"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:55:50.723880 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.723837 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-istio-envoy" pod "367229e1-f694-4449-9e91-eadaf2502938" (UID: "367229e1-f694-4449-9e91-eadaf2502938"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:50.724022 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.723964 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367229e1-f694-4449-9e91-eadaf2502938-istio-token" pod "367229e1-f694-4449-9e91-eadaf2502938" (UID: "367229e1-f694-4449-9e91-eadaf2502938"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:50.724114 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.724090 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367229e1-f694-4449-9e91-eadaf2502938-kube-api-access-5fqnt" pod "367229e1-f694-4449-9e91-eadaf2502938" (UID: "367229e1-f694-4449-9e91-eadaf2502938"). InnerVolumeSpecName "kube-api-access-5fqnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:50.822523 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.822478 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:50.822523 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.822514 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367229e1-f694-4449-9e91-eadaf2502938-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:50.822523 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.822524 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367229e1-f694-4449-9e91-eadaf2502938-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:50.822761 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.822535 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fqnt\" (UniqueName: \"kubernetes.io/projected/367229e1-f694-4449-9e91-eadaf2502938-kube-api-access-5fqnt\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:50.822761 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.822545 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:50.822761 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.822554 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:50.822761 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.822562 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367229e1-f694-4449-9e91-eadaf2502938-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:50.822761 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.822571 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:50.822761 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:50.822581 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367229e1-f694-4449-9e91-eadaf2502938-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:55:51.049215 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:51.049124 4433 generic.go:356] "Generic (PLEG): container finished" podID="367229e1-f694-4449-9e91-eadaf2502938" containerID="c034c45611b330f8e24a09b20f826204843342f85b43de54ff79a3f41715b897" exitCode=0 Oct 30 19:55:51.049215 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:51.049187 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" Oct 30 19:55:51.049403 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:51.049204 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" event={"ID":"367229e1-f694-4449-9e91-eadaf2502938","Type":"ContainerDied","Data":"c034c45611b330f8e24a09b20f826204843342f85b43de54ff79a3f41715b897"} Oct 30 19:55:51.049403 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:51.049264 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5" event={"ID":"367229e1-f694-4449-9e91-eadaf2502938","Type":"ContainerDied","Data":"5fa0a2cf2a548930f0877d7809412a2dbc97e0994bed16e1b83598808cf7dee0"} Oct 30 19:55:51.049403 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:51.049288 4433 scope.go:122] "RemoveContainer" containerID="c034c45611b330f8e24a09b20f826204843342f85b43de54ff79a3f41715b897" Oct 30 19:55:51.060893 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:51.060869 4433 scope.go:122] "RemoveContainer" containerID="c034c45611b330f8e24a09b20f826204843342f85b43de54ff79a3f41715b897" Oct 30 19:55:51.061209 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:55:51.061183 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c034c45611b330f8e24a09b20f826204843342f85b43de54ff79a3f41715b897\": container with ID starting with c034c45611b330f8e24a09b20f826204843342f85b43de54ff79a3f41715b897 not found: ID does not exist" containerID="c034c45611b330f8e24a09b20f826204843342f85b43de54ff79a3f41715b897" Oct 30 19:55:51.061292 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:51.061218 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c034c45611b330f8e24a09b20f826204843342f85b43de54ff79a3f41715b897"} err="failed to get container status \"c034c45611b330f8e24a09b20f826204843342f85b43de54ff79a3f41715b897\": rpc error: code = NotFound desc = could not find container \"c034c45611b330f8e24a09b20f826204843342f85b43de54ff79a3f41715b897\": container with ID starting with c034c45611b330f8e24a09b20f826204843342f85b43de54ff79a3f41715b897 not found: ID does not exist" Oct 30 19:55:51.217459 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:51.217429 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5"] Oct 30 19:55:51.218467 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:51.218445 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-47jn5"] Oct 30 19:55:51.307195 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:51.307098 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:51.556255 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:51.556214 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:51.783134 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:51.783092 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:53.217155 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:53.217123 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="367229e1-f694-4449-9e91-eadaf2502938" path="/var/lib/kubelet/pods/367229e1-f694-4449-9e91-eadaf2502938/volumes" Oct 30 19:55:53.366879 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:55:53.366843 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:55:56.347615 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:55:56.347579 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:01.113053 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:01.112964 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:05.104956 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:05.104920 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:05.334249 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:05.334219 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:06.090537 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:06.090492 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:11.203447 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:11.203410 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:13.684023 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:13.681382 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:13.910331 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:13.910287 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:14.823121 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:14.823077 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:15.050295 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:15.050244 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:16.464848 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:16.464816 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:17.076673 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:17.076418 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:17.378580 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:17.378462 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:20.268043 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:20.268002 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:20.660634 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:20.660592 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:20.888504 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:20.888470 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:21.234662 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:21.234626 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:23.290404 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:23.290324 4433 generic.go:356] "Generic (PLEG): container finished" podID="096a248d-1aed-493d-9ec1-99bf2a8217eb" containerID="6e66a979d6dd47eaf68da344e7f2c4e5c4cf5f5629edbe513d4a3451ecc01ed9" exitCode=0 Oct 30 19:56:23.290822 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:23.290403 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" event={"ID":"096a248d-1aed-493d-9ec1-99bf2a8217eb","Type":"ContainerDied","Data":"6e66a979d6dd47eaf68da344e7f2c4e5c4cf5f5629edbe513d4a3451ecc01ed9"} Oct 30 19:56:23.290870 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:23.290858 4433 scope.go:122] "RemoveContainer" containerID="6e66a979d6dd47eaf68da344e7f2c4e5c4cf5f5629edbe513d4a3451ecc01ed9" Oct 30 19:56:23.292661 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:23.292635 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:56:23.292724 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:23.292702 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:56:23.292760 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:23.292735 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:56:23.377862 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:56:23.377830 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:56:24.302038 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:24.302005 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" event={"ID":"096a248d-1aed-493d-9ec1-99bf2a8217eb","Type":"ContainerStarted","Data":"55bf821183c265f7ff5a2c706f4136c40d23125f896d6cb102ee915ea7faf3aa"} Oct 30 19:56:24.746703 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:24.746659 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:56:24.747973 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:24.747942 4433 patch_prober.go:29] interesting pod/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 19:56:24.748143 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:24.748043 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" podUID="096a248d-1aed-493d-9ec1-99bf2a8217eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 19:56:25.747038 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:25.746972 4433 patch_prober.go:29] interesting pod/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 19:56:25.747545 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:25.747058 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" podUID="096a248d-1aed-493d-9ec1-99bf2a8217eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 19:56:25.872483 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:25.872420 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:26.746317 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:26.746276 4433 patch_prober.go:29] interesting pod/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 19:56:26.746497 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:26.746339 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" podUID="096a248d-1aed-493d-9ec1-99bf2a8217eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 19:56:26.973421 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:26.973385 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:56:27.746674 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:27.746633 4433 patch_prober.go:29] interesting pod/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 19:56:27.746865 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:27.746696 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" podUID="096a248d-1aed-493d-9ec1-99bf2a8217eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 19:56:28.746383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:28.746343 4433 patch_prober.go:29] interesting pod/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 19:56:28.746798 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:28.746402 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" podUID="096a248d-1aed-493d-9ec1-99bf2a8217eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 19:56:29.746894 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:29.746856 4433 patch_prober.go:29] interesting pod/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 19:56:29.747350 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:29.746913 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" podUID="096a248d-1aed-493d-9ec1-99bf2a8217eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 19:56:30.751060 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:30.751023 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:56:30.752356 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:30.752332 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:56:30.860375 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:30.860337 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:36.111508 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:36.111470 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:38.481591 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:38.481543 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:38.725684 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:38.725652 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:41.222540 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:41.222505 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:42.529305 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:42.529262 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:42.776942 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:42.776902 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:44.650580 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:44.650544 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:46.058993 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:46.058947 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.093506 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:51.093434 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.330591 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:51.330553 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:53.367456 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:56:53.367413 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:56:56.153613 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:56:56.153579 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:00.411741 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:00.411706 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:01.900172 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:01.900143 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:02.220746 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:02.220670 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:02.455875 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:02.455846 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:02.877715 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:02.877689 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:07.206503 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:07.206406 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:09.393233 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:09.393198 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:09.658090 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:09.658018 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:09.927665 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:09.927571 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:12.270092 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:12.270046 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:17.407934 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:17.405025 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:17.855202 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:17.855166 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:18.664195 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:18.664153 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:19.045040 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:19.044947 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:20.598827 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:20.598774 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:21.962075 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:21.962036 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:22.574062 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:22.574008 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:22.815170 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:22.815128 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:23.042237 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:23.042148 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:23.265190 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:23.265155 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:23.367636 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:57:23.367597 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:57:23.711623 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:23.711509 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:23.933155 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:23.933120 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:26.395138 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:26.395103 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:26.690633 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:26.690549 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:26.994592 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:26.994512 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:27.338455 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:27.338420 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:27.806383 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:27.806304 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:30.884846 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:30.884806 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:31.118342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:31.118306 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:32.907616 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:32.907586 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:37.934949 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:37.934917 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:42.804590 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:42.804531 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:45.001818 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:45.001760 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:48.051821 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:48.051788 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:48.941738 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:48.941703 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:52.713260 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:52.713219 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:53.367043 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:57:53.367006 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:57:56.614903 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:56.614862 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g"] Oct 30 19:57:56.615399 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:56.615083 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" podUID="7a6f7dc5-3d1d-4b4b-b714-cded507fb414" containerName="maas-api" containerID="cri-o://1cdeaa658539870fc608231de49cb87a5aecc8b02cf9cb5106cff8d0e0ae84c5" gracePeriod=30 Oct 30 19:57:56.822599 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:56.822563 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:56.942370 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:56.942342 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:57:57.061270 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.061223 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-maas-api-tls\") pod \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " Oct 30 19:57:57.061484 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.061360 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-kube-api-access-lk82r\" (UniqueName: \"kubernetes.io/projected/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-kube-api-access-lk82r\") pod \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " Oct 30 19:57:57.061579 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.061547 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-metrics-certs\") pod \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " Oct 30 19:57:57.061675 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.061647 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-openshift-service-ca\") pod \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\" (UID: \"7a6f7dc5-3d1d-4b4b-b714-cded507fb414\") " Oct 30 19:57:57.062587 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.062553 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-openshift-service-ca" pod "7a6f7dc5-3d1d-4b4b-b714-cded507fb414" (UID: "7a6f7dc5-3d1d-4b4b-b714-cded507fb414"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:57.066174 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.066142 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-maas-api-tls" pod "7a6f7dc5-3d1d-4b4b-b714-cded507fb414" (UID: "7a6f7dc5-3d1d-4b4b-b714-cded507fb414"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:57.066600 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.066574 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-metrics-certs" pod "7a6f7dc5-3d1d-4b4b-b714-cded507fb414" (UID: "7a6f7dc5-3d1d-4b4b-b714-cded507fb414"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:57.067252 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.067224 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-kube-api-access-lk82r" pod "7a6f7dc5-3d1d-4b4b-b714-cded507fb414" (UID: "7a6f7dc5-3d1d-4b4b-b714-cded507fb414"). InnerVolumeSpecName "kube-api-access-lk82r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:57.111818 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.111770 4433 generic.go:356] "Generic (PLEG): container finished" podID="7a6f7dc5-3d1d-4b4b-b714-cded507fb414" containerID="1cdeaa658539870fc608231de49cb87a5aecc8b02cf9cb5106cff8d0e0ae84c5" exitCode=0 Oct 30 19:57:57.112085 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.112028 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" event={"ID":"7a6f7dc5-3d1d-4b4b-b714-cded507fb414","Type":"ContainerDied","Data":"1cdeaa658539870fc608231de49cb87a5aecc8b02cf9cb5106cff8d0e0ae84c5"} Oct 30 19:57:57.112085 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.112082 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" event={"ID":"7a6f7dc5-3d1d-4b4b-b714-cded507fb414","Type":"ContainerDied","Data":"8570c8e8b360368e0707b4ef1638a1488b2015aa66a7a01fddc330aa3e0bd44f"} Oct 30 19:57:57.112236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.112110 4433 scope.go:122] "RemoveContainer" containerID="1cdeaa658539870fc608231de49cb87a5aecc8b02cf9cb5106cff8d0e0ae84c5" Oct 30 19:57:57.112296 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.112266 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g" Oct 30 19:57:57.127855 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.127831 4433 scope.go:122] "RemoveContainer" containerID="1cdeaa658539870fc608231de49cb87a5aecc8b02cf9cb5106cff8d0e0ae84c5" Oct 30 19:57:57.128194 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:57:57.128171 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cdeaa658539870fc608231de49cb87a5aecc8b02cf9cb5106cff8d0e0ae84c5\": container with ID starting with 1cdeaa658539870fc608231de49cb87a5aecc8b02cf9cb5106cff8d0e0ae84c5 not found: ID does not exist" containerID="1cdeaa658539870fc608231de49cb87a5aecc8b02cf9cb5106cff8d0e0ae84c5" Oct 30 19:57:57.128246 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.128204 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cdeaa658539870fc608231de49cb87a5aecc8b02cf9cb5106cff8d0e0ae84c5"} err="failed to get container status \"1cdeaa658539870fc608231de49cb87a5aecc8b02cf9cb5106cff8d0e0ae84c5\": rpc error: code = NotFound desc = could not find container \"1cdeaa658539870fc608231de49cb87a5aecc8b02cf9cb5106cff8d0e0ae84c5\": container with ID starting with 1cdeaa658539870fc608231de49cb87a5aecc8b02cf9cb5106cff8d0e0ae84c5 not found: ID does not exist" Oct 30 19:57:57.163173 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.163089 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-metrics-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:57:57.163173 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.163118 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-openshift-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:57:57.163173 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.163129 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-maas-api-tls\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:57:57.163173 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.163137 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk82r\" (UniqueName: \"kubernetes.io/projected/7a6f7dc5-3d1d-4b4b-b714-cded507fb414-kube-api-access-lk82r\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:57:57.284898 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.284853 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g"] Oct 30 19:57:57.287301 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:57.287278 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-898968-644fb6f85f-sbk2g"] Oct 30 19:57:58.329649 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:58.329613 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:58.644193 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:58.644098 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:58.902561 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:58.902467 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:59.216597 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:59.216509 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a6f7dc5-3d1d-4b4b-b714-cded507fb414" path="/var/lib/kubelet/pods/7a6f7dc5-3d1d-4b4b-b714-cded507fb414/volumes" Oct 30 19:57:59.378955 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:57:59.378919 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:03.012436 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:03.012395 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:04.680920 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:04.680884 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:04.976245 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:04.976138 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:05.198779 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:05.198743 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:05.485908 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:05.485876 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:05.719710 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:05.719674 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:05.961808 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:05.961765 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:07.803554 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:07.803514 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:10.135364 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:10.135319 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:10.410645 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:10.410562 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:12.259356 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:12.259322 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:12.517456 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:12.517370 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:12.803398 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:12.803309 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:13.047448 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:13.047417 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:15.238160 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:15.238124 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:19.731140 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:19.731104 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft"] Oct 30 19:58:19.844829 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:19.844294 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh"] Oct 30 19:58:20.195686 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:20.195646 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:20.441729 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:20.441696 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:23.365889 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:58:23.365851 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:58:24.731637 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:24.731590 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" podUID="bb823cde-6a63-4fcd-acc7-4700d65ba7ac" containerName="payload-processing" containerID="cri-o://0c127ed7314151428769ac31cc3b831da26cb96ce74beb382a4debb5a8c7c0cb" gracePeriod=25 Oct 30 19:58:24.845437 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:24.845389 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" podUID="4026634f-e0b5-4b64-a0ce-ebfe65c07365" containerName="payload-pre-processing" containerID="cri-o://6d43f4b52e7b7ca99cc7ca6ee6c337c411c8721dcacbae58e8a400fefd64ab6d" gracePeriod=25 Oct 30 19:58:25.025558 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:25.025464 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:26.339113 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:26.339084 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:30.352013 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:30.351958 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:33.297737 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:33.297702 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" podUID="bb823cde-6a63-4fcd-acc7-4700d65ba7ac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:33.297737 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:33.297723 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" podUID="4026634f-e0b5-4b64-a0ce-ebfe65c07365" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:34.733523 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:34.733444 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:35.317558 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:35.317520 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:35.613199 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:35.613096 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:36.587291 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:36.587263 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:36.974180 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:36.974099 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:37.489149 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:37.489112 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:39.136955 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.136920 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb"] Oct 30 19:58:39.137577 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.137560 4433 cpu_manager.go:414] "containerMap: removing container" podUID="7a6f7dc5-3d1d-4b4b-b714-cded507fb414" containerName="maas-api" Oct 30 19:58:39.137626 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.137578 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a6f7dc5-3d1d-4b4b-b714-cded507fb414" containerName="maas-api" Oct 30 19:58:39.137626 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.137602 4433 cpu_manager.go:414] "containerMap: removing container" podUID="335bb767-624e-44ba-8dab-3bab19297c9e" containerName="istio-proxy" Oct 30 19:58:39.137626 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.137607 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335bb767-624e-44ba-8dab-3bab19297c9e" containerName="istio-proxy" Oct 30 19:58:39.137626 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.137620 4433 cpu_manager.go:414] "containerMap: removing container" podUID="34bbe99d-caf6-46cb-a4b4-622ee14e8d75" containerName="istio-proxy" Oct 30 19:58:39.137626 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.137626 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34bbe99d-caf6-46cb-a4b4-622ee14e8d75" containerName="istio-proxy" Oct 30 19:58:39.137781 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.137634 4433 cpu_manager.go:414] "containerMap: removing container" podUID="367229e1-f694-4449-9e91-eadaf2502938" containerName="istio-proxy" Oct 30 19:58:39.137781 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.137638 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367229e1-f694-4449-9e91-eadaf2502938" containerName="istio-proxy" Oct 30 19:58:39.137781 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.137657 4433 cpu_manager.go:414] "containerMap: removing container" podUID="abe0d869-8b60-4ef3-acf4-8a0c2375b35c" containerName="istio-proxy" Oct 30 19:58:39.137781 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.137661 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe0d869-8b60-4ef3-acf4-8a0c2375b35c" containerName="istio-proxy" Oct 30 19:58:39.137781 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.137755 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a6f7dc5-3d1d-4b4b-b714-cded507fb414" containerName="maas-api" Oct 30 19:58:39.137781 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.137767 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="335bb767-624e-44ba-8dab-3bab19297c9e" containerName="istio-proxy" Oct 30 19:58:39.137781 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.137774 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="34bbe99d-caf6-46cb-a4b4-622ee14e8d75" containerName="istio-proxy" Oct 30 19:58:39.137781 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.137780 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="367229e1-f694-4449-9e91-eadaf2502938" containerName="istio-proxy" Oct 30 19:58:39.138052 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.137787 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe0d869-8b60-4ef3-acf4-8a0c2375b35c" containerName="istio-proxy" Oct 30 19:58:39.141370 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.141346 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.247607 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.247572 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb"] Oct 30 19:58:39.272551 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.272513 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1550c04a-cebb-456c-9a05-4eeddaefef8a-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.273015 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.272969 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.273169 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.273047 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.273169 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.273077 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnxxx\" (UniqueName: \"kubernetes.io/projected/1550c04a-cebb-456c-9a05-4eeddaefef8a-kube-api-access-hnxxx\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.273310 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.273265 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.273368 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.273345 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.273448 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.273420 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.273508 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.273484 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.273565 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.273520 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.375406 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.375354 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.375594 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.375415 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.375594 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.375460 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.375594 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.375482 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.375594 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.375547 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1550c04a-cebb-456c-9a05-4eeddaefef8a-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.375594 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.375592 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.375783 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.375618 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.375783 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.375638 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnxxx\" (UniqueName: \"kubernetes.io/projected/1550c04a-cebb-456c-9a05-4eeddaefef8a-kube-api-access-hnxxx\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.375783 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.375759 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.376376 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.376349 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.376679 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.376658 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.376782 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.376717 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1550c04a-cebb-456c-9a05-4eeddaefef8a-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.376928 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.376899 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.377069 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.377051 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.378712 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.378689 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.378900 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.378881 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.432965 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.432901 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.433236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.433212 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnxxx\" (UniqueName: \"kubernetes.io/projected/1550c04a-cebb-456c-9a05-4eeddaefef8a-kube-api-access-hnxxx\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.454511 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.454481 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:39.890355 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.890325 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb"] Oct 30 19:58:39.891736 ip-10-0-141-5 kubenswrapper[4433]: W1030 19:58:39.891707 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1550c04a_cebb_456c_9a05_4eeddaefef8a.slice/crio-03186d34b1561d58961a1f1f236ddc0fefbe4b62a94d015782f928f380e17ec3 WatchSource:0}: Error finding container 03186d34b1561d58961a1f1f236ddc0fefbe4b62a94d015782f928f380e17ec3: Status 404 returned error can't find the container with id 03186d34b1561d58961a1f1f236ddc0fefbe4b62a94d015782f928f380e17ec3 Oct 30 19:58:39.894935 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.894901 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:39.895057 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.895014 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:39.895124 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:39.895065 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:40.533417 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:40.533371 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" event={"ID":"1550c04a-cebb-456c-9a05-4eeddaefef8a","Type":"ContainerStarted","Data":"c25dd67b3fc028d75690d134d0eb567ff717e7cf448facf4d65aaccb41e5e3ee"} Oct 30 19:58:40.533925 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:40.533468 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" event={"ID":"1550c04a-cebb-456c-9a05-4eeddaefef8a","Type":"ContainerStarted","Data":"03186d34b1561d58961a1f1f236ddc0fefbe4b62a94d015782f928f380e17ec3"} Oct 30 19:58:40.674606 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:40.674552 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" podStartSLOduration=1.67453824 podStartE2EDuration="1.67453824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:40.670308519 +0000 UTC m=+2567.605096043" watchObservedRunningTime="2026-10-30 19:58:40.67453824 +0000 UTC m=+2567.609325763" Oct 30 19:58:41.454815 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:41.454778 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:41.456478 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:41.456449 4433 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:58:41.456619 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:41.456503 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" podUID="1550c04a-cebb-456c-9a05-4eeddaefef8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:58:42.317555 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:42.317511 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:42.455201 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:42.455166 4433 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:58:42.455348 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:42.455228 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" podUID="1550c04a-cebb-456c-9a05-4eeddaefef8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:58:43.309933 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:43.309879 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" podUID="4026634f-e0b5-4b64-a0ce-ebfe65c07365" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:43.310174 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:43.309957 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" podUID="bb823cde-6a63-4fcd-acc7-4700d65ba7ac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:43.454858 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:43.454823 4433 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:58:43.455266 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:43.454876 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" podUID="1550c04a-cebb-456c-9a05-4eeddaefef8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:58:43.850017 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:43.849944 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd"] Oct 30 19:58:43.850573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:43.850539 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" podUID="666f2ff7-bd9c-4525-bdf9-bda82db33b9a" containerName="istio-proxy" containerID="cri-o://5a957cc813899e54929c9ab214df6186723d60c904984bf84904ce57d5dee561" gracePeriod=30 Oct 30 19:58:43.866227 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:43.866165 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m"] Oct 30 19:58:43.866523 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:43.866475 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" podUID="5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" containerName="istio-proxy" containerID="cri-o://c12f10c5298f6bda829aae5abe4395c7c9d153f47c83ae85f679ed7317396c03" gracePeriod=30 Oct 30 19:58:44.455360 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:44.455314 4433 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:58:44.455741 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:44.455381 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" podUID="1550c04a-cebb-456c-9a05-4eeddaefef8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:58:45.053798 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.053770 4433 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:58:45.053971 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.053815 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" podUID="5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:58:45.054175 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.054136 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" Oct 30 19:58:45.111003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.109502 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb823cde-6a63-4fcd-acc7-4700d65ba7ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb823cde-6a63-4fcd-acc7-4700d65ba7ac-plugins-config-volume\") pod \"bb823cde-6a63-4fcd-acc7-4700d65ba7ac\" (UID: \"bb823cde-6a63-4fcd-acc7-4700d65ba7ac\") " Oct 30 19:58:45.111003 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.109693 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb823cde-6a63-4fcd-acc7-4700d65ba7ac-kube-api-access-9lxbc\" (UniqueName: \"kubernetes.io/projected/bb823cde-6a63-4fcd-acc7-4700d65ba7ac-kube-api-access-9lxbc\") pod \"bb823cde-6a63-4fcd-acc7-4700d65ba7ac\" (UID: \"bb823cde-6a63-4fcd-acc7-4700d65ba7ac\") " Oct 30 19:58:45.113388 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.113075 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb823cde-6a63-4fcd-acc7-4700d65ba7ac-plugins-config-volume" pod "bb823cde-6a63-4fcd-acc7-4700d65ba7ac" (UID: "bb823cde-6a63-4fcd-acc7-4700d65ba7ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:45.116315 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.116288 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb823cde-6a63-4fcd-acc7-4700d65ba7ac-kube-api-access-9lxbc" pod "bb823cde-6a63-4fcd-acc7-4700d65ba7ac" (UID: "bb823cde-6a63-4fcd-acc7-4700d65ba7ac"). InnerVolumeSpecName "kube-api-access-9lxbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:45.167517 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.167491 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" Oct 30 19:58:45.212316 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.212278 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lxbc\" (UniqueName: \"kubernetes.io/projected/bb823cde-6a63-4fcd-acc7-4700d65ba7ac-kube-api-access-9lxbc\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:45.212316 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.212318 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb823cde-6a63-4fcd-acc7-4700d65ba7ac-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:45.314204 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.312921 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4026634f-e0b5-4b64-a0ce-ebfe65c07365-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4026634f-e0b5-4b64-a0ce-ebfe65c07365-plugins-config-volume\") pod \"4026634f-e0b5-4b64-a0ce-ebfe65c07365\" (UID: \"4026634f-e0b5-4b64-a0ce-ebfe65c07365\") " Oct 30 19:58:45.314204 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.313508 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4026634f-e0b5-4b64-a0ce-ebfe65c07365-kube-api-access-qqfzp\" (UniqueName: \"kubernetes.io/projected/4026634f-e0b5-4b64-a0ce-ebfe65c07365-kube-api-access-qqfzp\") pod \"4026634f-e0b5-4b64-a0ce-ebfe65c07365\" (UID: \"4026634f-e0b5-4b64-a0ce-ebfe65c07365\") " Oct 30 19:58:45.314204 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.313796 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4026634f-e0b5-4b64-a0ce-ebfe65c07365-plugins-config-volume" pod "4026634f-e0b5-4b64-a0ce-ebfe65c07365" (UID: "4026634f-e0b5-4b64-a0ce-ebfe65c07365"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:45.314579 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.314555 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4026634f-e0b5-4b64-a0ce-ebfe65c07365-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:45.316500 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.316467 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4026634f-e0b5-4b64-a0ce-ebfe65c07365-kube-api-access-qqfzp" pod "4026634f-e0b5-4b64-a0ce-ebfe65c07365" (UID: "4026634f-e0b5-4b64-a0ce-ebfe65c07365"). InnerVolumeSpecName "kube-api-access-qqfzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:45.370161 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.370134 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:45.415962 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.415929 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqfzp\" (UniqueName: \"kubernetes.io/projected/4026634f-e0b5-4b64-a0ce-ebfe65c07365-kube-api-access-qqfzp\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:45.454786 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.454753 4433 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:58:45.454916 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.454822 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" podUID="1550c04a-cebb-456c-9a05-4eeddaefef8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:58:45.569448 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.569370 4433 generic.go:356] "Generic (PLEG): container finished" podID="bb823cde-6a63-4fcd-acc7-4700d65ba7ac" containerID="0c127ed7314151428769ac31cc3b831da26cb96ce74beb382a4debb5a8c7c0cb" exitCode=0 Oct 30 19:58:45.569448 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.569417 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" Oct 30 19:58:45.569925 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.569445 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" event={"ID":"bb823cde-6a63-4fcd-acc7-4700d65ba7ac","Type":"ContainerDied","Data":"0c127ed7314151428769ac31cc3b831da26cb96ce74beb382a4debb5a8c7c0cb"} Oct 30 19:58:45.569925 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.569494 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft" event={"ID":"bb823cde-6a63-4fcd-acc7-4700d65ba7ac","Type":"ContainerDied","Data":"9d8fbcf7407fd635aa33d1a7a6f953d923cd2e2412b3e8545d6214401224c89d"} Oct 30 19:58:45.569925 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.569514 4433 scope.go:122] "RemoveContainer" containerID="0c127ed7314151428769ac31cc3b831da26cb96ce74beb382a4debb5a8c7c0cb" Oct 30 19:58:45.571057 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.571033 4433 generic.go:356] "Generic (PLEG): container finished" podID="4026634f-e0b5-4b64-a0ce-ebfe65c07365" containerID="6d43f4b52e7b7ca99cc7ca6ee6c337c411c8721dcacbae58e8a400fefd64ab6d" exitCode=0 Oct 30 19:58:45.571179 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.571070 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" Oct 30 19:58:45.571179 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.571165 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" event={"ID":"4026634f-e0b5-4b64-a0ce-ebfe65c07365","Type":"ContainerDied","Data":"6d43f4b52e7b7ca99cc7ca6ee6c337c411c8721dcacbae58e8a400fefd64ab6d"} Oct 30 19:58:45.571303 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.571207 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh" event={"ID":"4026634f-e0b5-4b64-a0ce-ebfe65c07365","Type":"ContainerDied","Data":"6db5e2cdb5964aeb26ee4c4bfdf7f56e7fc15fea0455fd69e2adb2f04226cdbd"} Oct 30 19:58:45.579813 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.579797 4433 scope.go:122] "RemoveContainer" containerID="0c127ed7314151428769ac31cc3b831da26cb96ce74beb382a4debb5a8c7c0cb" Oct 30 19:58:45.580094 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:58:45.580072 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c127ed7314151428769ac31cc3b831da26cb96ce74beb382a4debb5a8c7c0cb\": container with ID starting with 0c127ed7314151428769ac31cc3b831da26cb96ce74beb382a4debb5a8c7c0cb not found: ID does not exist" containerID="0c127ed7314151428769ac31cc3b831da26cb96ce74beb382a4debb5a8c7c0cb" Oct 30 19:58:45.580155 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.580101 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c127ed7314151428769ac31cc3b831da26cb96ce74beb382a4debb5a8c7c0cb"} err="failed to get container status \"0c127ed7314151428769ac31cc3b831da26cb96ce74beb382a4debb5a8c7c0cb\": rpc error: code = NotFound desc = could not find container \"0c127ed7314151428769ac31cc3b831da26cb96ce74beb382a4debb5a8c7c0cb\": container with ID starting with 0c127ed7314151428769ac31cc3b831da26cb96ce74beb382a4debb5a8c7c0cb not found: ID does not exist" Oct 30 19:58:45.580155 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.580117 4433 scope.go:122] "RemoveContainer" containerID="6d43f4b52e7b7ca99cc7ca6ee6c337c411c8721dcacbae58e8a400fefd64ab6d" Oct 30 19:58:45.615580 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.615560 4433 scope.go:122] "RemoveContainer" containerID="6d43f4b52e7b7ca99cc7ca6ee6c337c411c8721dcacbae58e8a400fefd64ab6d" Oct 30 19:58:45.615900 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:58:45.615880 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d43f4b52e7b7ca99cc7ca6ee6c337c411c8721dcacbae58e8a400fefd64ab6d\": container with ID starting with 6d43f4b52e7b7ca99cc7ca6ee6c337c411c8721dcacbae58e8a400fefd64ab6d not found: ID does not exist" containerID="6d43f4b52e7b7ca99cc7ca6ee6c337c411c8721dcacbae58e8a400fefd64ab6d" Oct 30 19:58:45.615969 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.615906 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d43f4b52e7b7ca99cc7ca6ee6c337c411c8721dcacbae58e8a400fefd64ab6d"} err="failed to get container status \"6d43f4b52e7b7ca99cc7ca6ee6c337c411c8721dcacbae58e8a400fefd64ab6d\": rpc error: code = NotFound desc = could not find container \"6d43f4b52e7b7ca99cc7ca6ee6c337c411c8721dcacbae58e8a400fefd64ab6d\": container with ID starting with 6d43f4b52e7b7ca99cc7ca6ee6c337c411c8721dcacbae58e8a400fefd64ab6d not found: ID does not exist" Oct 30 19:58:45.757163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.757133 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh"] Oct 30 19:58:45.775529 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.775496 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d52df0-5844f7bcc5-5drwh"] Oct 30 19:58:45.884662 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.884602 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft"] Oct 30 19:58:45.888811 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:45.888789 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d52df0-594f69d78b-c9mft"] Oct 30 19:58:46.199491 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:46.199408 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:46.461022 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:46.460926 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:46.461360 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:46.461342 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:46.462135 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:46.462116 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 19:58:46.476438 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:46.476414 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:46.801596 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:46.801511 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:47.216934 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:47.216895 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4026634f-e0b5-4b64-a0ce-ebfe65c07365" path="/var/lib/kubelet/pods/4026634f-e0b5-4b64-a0ce-ebfe65c07365/volumes" Oct 30 19:58:47.217636 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:47.217614 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb823cde-6a63-4fcd-acc7-4700d65ba7ac" path="/var/lib/kubelet/pods/bb823cde-6a63-4fcd-acc7-4700d65ba7ac/volumes" Oct 30 19:58:48.052155 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:48.052118 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:49.150723 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.150656 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:58:49.194765 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.194737 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:58:49.288846 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.283372 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-credential-socket\") pod \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " Oct 30 19:58:49.288846 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.283444 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-workload-certs\") pod \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " Oct 30 19:58:49.288846 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.283479 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-data\") pod \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " Oct 30 19:58:49.288846 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.283536 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istiod-ca-cert\") pod \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " Oct 30 19:58:49.288846 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.284406 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-data" pod "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" (UID: "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:49.288846 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.284671 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-credential-socket" pod "666f2ff7-bd9c-4525-bdf9-bda82db33b9a" (UID: "666f2ff7-bd9c-4525-bdf9-bda82db33b9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:49.288846 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.285111 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istiod-ca-cert" pod "666f2ff7-bd9c-4525-bdf9-bda82db33b9a" (UID: "666f2ff7-bd9c-4525-bdf9-bda82db33b9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:49.288846 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.285192 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-workload-certs" pod "666f2ff7-bd9c-4525-bdf9-bda82db33b9a" (UID: "666f2ff7-bd9c-4525-bdf9-bda82db33b9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:49.297104 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.297060 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-workload-socket\") pod \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " Oct 30 19:58:49.297297 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.297136 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-data\") pod \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " Oct 30 19:58:49.297297 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.297175 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-workload-socket\") pod \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " Oct 30 19:58:49.297297 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.297228 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-envoy\") pod \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " Oct 30 19:58:49.297552 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.297300 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-podinfo\") pod \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " Oct 30 19:58:49.297552 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.297305 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-workload-socket" pod "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" (UID: "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:49.297552 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.297342 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-credential-socket\") pod \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " Oct 30 19:58:49.297552 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.297416 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-data" pod "666f2ff7-bd9c-4525-bdf9-bda82db33b9a" (UID: "666f2ff7-bd9c-4525-bdf9-bda82db33b9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:49.297552 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.297430 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istiod-ca-cert\") pod \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " Oct 30 19:58:49.297552 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.297463 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-envoy\") pod \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " Oct 30 19:58:49.297871 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.297589 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-workload-socket" pod "666f2ff7-bd9c-4525-bdf9-bda82db33b9a" (UID: "666f2ff7-bd9c-4525-bdf9-bda82db33b9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:49.298458 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.297958 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-credential-socket" pod "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" (UID: "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:49.298458 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.298052 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-kube-api-access-pp2g2\" (UniqueName: \"kubernetes.io/projected/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-kube-api-access-pp2g2\") pod \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " Oct 30 19:58:49.298458 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.298079 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-kube-api-access-svf2w\" (UniqueName: \"kubernetes.io/projected/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-kube-api-access-svf2w\") pod \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " Oct 30 19:58:49.298458 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.298114 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-token\" (UniqueName: \"kubernetes.io/projected/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-token\") pod \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " Oct 30 19:58:49.298458 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.298244 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-podinfo\") pod \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " Oct 30 19:58:49.298458 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.298316 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-workload-certs\") pod \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\" (UID: \"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a\") " Oct 30 19:58:49.298458 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.298372 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-token\" (UniqueName: \"kubernetes.io/projected/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-token\") pod \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\" (UID: \"666f2ff7-bd9c-4525-bdf9-bda82db33b9a\") " Oct 30 19:58:49.301028 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.299299 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.301028 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.299334 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.301028 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.299349 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.301028 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.299909 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istiod-ca-cert" pod "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" (UID: "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:49.301028 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.300681 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-kube-api-access-pp2g2" pod "666f2ff7-bd9c-4525-bdf9-bda82db33b9a" (UID: "666f2ff7-bd9c-4525-bdf9-bda82db33b9a"). InnerVolumeSpecName "kube-api-access-pp2g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:49.301344 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.301206 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-envoy" pod "666f2ff7-bd9c-4525-bdf9-bda82db33b9a" (UID: "666f2ff7-bd9c-4525-bdf9-bda82db33b9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:49.301474 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.301426 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.301552 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.301477 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-workload-certs" pod "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" (UID: "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:49.302236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.301682 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.302236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.301713 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.302236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.301729 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.302236 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.301743 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.302633 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.302589 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-podinfo" pod "666f2ff7-bd9c-4525-bdf9-bda82db33b9a" (UID: "666f2ff7-bd9c-4525-bdf9-bda82db33b9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:49.302962 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.302938 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-token" pod "666f2ff7-bd9c-4525-bdf9-bda82db33b9a" (UID: "666f2ff7-bd9c-4525-bdf9-bda82db33b9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:49.303393 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.303368 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-token" pod "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" (UID: "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:49.303476 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.303415 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-kube-api-access-svf2w" pod "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" (UID: "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a"). InnerVolumeSpecName "kube-api-access-svf2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:49.303530 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.303505 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-podinfo" pod "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" (UID: "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:49.304280 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.304259 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-envoy" pod "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" (UID: "5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:49.317507 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.317472 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:49.403796 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.403722 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.403796 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.403748 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.403796 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.403760 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.403796 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.403773 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.403796 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.403781 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.403796 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.403790 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.403796 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.403798 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.404124 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.403807 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp2g2\" (UniqueName: \"kubernetes.io/projected/666f2ff7-bd9c-4525-bdf9-bda82db33b9a-kube-api-access-pp2g2\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.404124 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.403817 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svf2w\" (UniqueName: \"kubernetes.io/projected/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-kube-api-access-svf2w\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.404124 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.403825 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:49.667304 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.667222 4433 generic.go:356] "Generic (PLEG): container finished" podID="666f2ff7-bd9c-4525-bdf9-bda82db33b9a" containerID="5a957cc813899e54929c9ab214df6186723d60c904984bf84904ce57d5dee561" exitCode=0 Oct 30 19:58:49.667304 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.667274 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" Oct 30 19:58:49.667508 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.667303 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" event={"ID":"666f2ff7-bd9c-4525-bdf9-bda82db33b9a","Type":"ContainerDied","Data":"5a957cc813899e54929c9ab214df6186723d60c904984bf84904ce57d5dee561"} Oct 30 19:58:49.667508 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.667358 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd" event={"ID":"666f2ff7-bd9c-4525-bdf9-bda82db33b9a","Type":"ContainerDied","Data":"dacd04ae61162905ac6d055bf05a763a1f3999860ee054341ee2eacf72c5f9eb"} Oct 30 19:58:49.667508 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.667387 4433 scope.go:122] "RemoveContainer" containerID="5a957cc813899e54929c9ab214df6186723d60c904984bf84904ce57d5dee561" Oct 30 19:58:49.668802 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.668772 4433 generic.go:356] "Generic (PLEG): container finished" podID="5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" containerID="c12f10c5298f6bda829aae5abe4395c7c9d153f47c83ae85f679ed7317396c03" exitCode=0 Oct 30 19:58:49.668890 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.668814 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" Oct 30 19:58:49.668890 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.668828 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" event={"ID":"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a","Type":"ContainerDied","Data":"c12f10c5298f6bda829aae5abe4395c7c9d153f47c83ae85f679ed7317396c03"} Oct 30 19:58:49.668890 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.668884 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m" event={"ID":"5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a","Type":"ContainerDied","Data":"1d48ab511f92bad0fbde10d88ee38658191c0973e1b7c4d5ef406f609a02fe56"} Oct 30 19:58:49.678668 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.678593 4433 scope.go:122] "RemoveContainer" containerID="5a957cc813899e54929c9ab214df6186723d60c904984bf84904ce57d5dee561" Oct 30 19:58:49.678888 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:58:49.678867 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a957cc813899e54929c9ab214df6186723d60c904984bf84904ce57d5dee561\": container with ID starting with 5a957cc813899e54929c9ab214df6186723d60c904984bf84904ce57d5dee561 not found: ID does not exist" containerID="5a957cc813899e54929c9ab214df6186723d60c904984bf84904ce57d5dee561" Oct 30 19:58:49.678961 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.678898 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a957cc813899e54929c9ab214df6186723d60c904984bf84904ce57d5dee561"} err="failed to get container status \"5a957cc813899e54929c9ab214df6186723d60c904984bf84904ce57d5dee561\": rpc error: code = NotFound desc = could not find container \"5a957cc813899e54929c9ab214df6186723d60c904984bf84904ce57d5dee561\": container with ID starting with 5a957cc813899e54929c9ab214df6186723d60c904984bf84904ce57d5dee561 not found: ID does not exist" Oct 30 19:58:49.678961 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.678914 4433 scope.go:122] "RemoveContainer" containerID="c12f10c5298f6bda829aae5abe4395c7c9d153f47c83ae85f679ed7317396c03" Oct 30 19:58:49.687910 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.687892 4433 scope.go:122] "RemoveContainer" containerID="c12f10c5298f6bda829aae5abe4395c7c9d153f47c83ae85f679ed7317396c03" Oct 30 19:58:49.688178 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:58:49.688160 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c12f10c5298f6bda829aae5abe4395c7c9d153f47c83ae85f679ed7317396c03\": container with ID starting with c12f10c5298f6bda829aae5abe4395c7c9d153f47c83ae85f679ed7317396c03 not found: ID does not exist" containerID="c12f10c5298f6bda829aae5abe4395c7c9d153f47c83ae85f679ed7317396c03" Oct 30 19:58:49.688246 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.688185 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12f10c5298f6bda829aae5abe4395c7c9d153f47c83ae85f679ed7317396c03"} err="failed to get container status \"c12f10c5298f6bda829aae5abe4395c7c9d153f47c83ae85f679ed7317396c03\": rpc error: code = NotFound desc = could not find container \"c12f10c5298f6bda829aae5abe4395c7c9d153f47c83ae85f679ed7317396c03\": container with ID starting with c12f10c5298f6bda829aae5abe4395c7c9d153f47c83ae85f679ed7317396c03 not found: ID does not exist" Oct 30 19:58:49.846585 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.846549 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m"] Oct 30 19:58:49.854314 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.854283 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9w98m"] Oct 30 19:58:49.957023 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:49.956918 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:50.005095 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:50.005051 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd"] Oct 30 19:58:50.005356 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:50.005115 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-5brxd"] Oct 30 19:58:50.280394 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:50.280308 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:51.215024 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:51.214972 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" path="/var/lib/kubelet/pods/5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a/volumes" Oct 30 19:58:51.215475 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:51.215461 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="666f2ff7-bd9c-4525-bdf9-bda82db33b9a" path="/var/lib/kubelet/pods/666f2ff7-bd9c-4525-bdf9-bda82db33b9a/volumes" Oct 30 19:58:52.211753 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:52.211721 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:53.314560 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:53.314528 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:53.371238 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:58:53.371196 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:58:53.549839 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:53.549809 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:53.814556 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:53.814532 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:54.405712 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:54.405685 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f"] Oct 30 19:58:54.406165 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:54.405874 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" podUID="a9420be3-395c-4e5a-b565-001d9b8cf7f6" containerName="istio-proxy" containerID="cri-o://25e2fe728004d5f187a614801b42224d9d87b9ad12dfda671bad5cc07f2986aa" gracePeriod=30 Oct 30 19:58:57.860284 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:57.860252 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:58.191963 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:58.191870 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dff2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:58:58.192438 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:58.192394 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" podUID="a9420be3-395c-4e5a-b565-001d9b8cf7f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:58:58.642648 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:58.642554 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:58.663509 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:58.663475 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6"] Oct 30 19:58:58.916814 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:58.916736 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:59.163797 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.163764 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:59.411483 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.411448 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:59.715098 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.715075 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:58:59.749901 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.749874 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:59.759804 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.759780 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-credential-socket\") pod \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " Oct 30 19:58:59.759910 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.759895 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-workload-socket\") pod \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " Oct 30 19:58:59.759950 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.759939 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-workload-certs\") pod \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " Oct 30 19:58:59.760057 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.760039 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-token\" (UniqueName: \"kubernetes.io/projected/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-token\") pod \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " Oct 30 19:58:59.760119 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.760075 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-envoy\") pod \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " Oct 30 19:58:59.760119 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.760075 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-credential-socket" pod "a9420be3-395c-4e5a-b565-001d9b8cf7f6" (UID: "a9420be3-395c-4e5a-b565-001d9b8cf7f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:59.760225 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.760125 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-data\") pod \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " Oct 30 19:58:59.760225 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.760181 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-podinfo\") pod \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " Oct 30 19:58:59.760225 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.760200 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-workload-socket" pod "a9420be3-395c-4e5a-b565-001d9b8cf7f6" (UID: "a9420be3-395c-4e5a-b565-001d9b8cf7f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:59.760370 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.760249 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9420be3-395c-4e5a-b565-001d9b8cf7f6-kube-api-access-rfgpk\" (UniqueName: \"kubernetes.io/projected/a9420be3-395c-4e5a-b565-001d9b8cf7f6-kube-api-access-rfgpk\") pod \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " Oct 30 19:58:59.760370 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.760284 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-workload-certs" pod "a9420be3-395c-4e5a-b565-001d9b8cf7f6" (UID: "a9420be3-395c-4e5a-b565-001d9b8cf7f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:59.760370 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.760322 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istiod-ca-cert\") pod \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\" (UID: \"a9420be3-395c-4e5a-b565-001d9b8cf7f6\") " Oct 30 19:58:59.760574 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.760544 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-data" pod "a9420be3-395c-4e5a-b565-001d9b8cf7f6" (UID: "a9420be3-395c-4e5a-b565-001d9b8cf7f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:59.760856 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.760833 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istiod-ca-cert" pod "a9420be3-395c-4e5a-b565-001d9b8cf7f6" (UID: "a9420be3-395c-4e5a-b565-001d9b8cf7f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:59.760956 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.760859 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.760956 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.760881 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.760956 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.760896 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.760956 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.760909 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.761826 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.761797 4433 generic.go:356] "Generic (PLEG): container finished" podID="a9420be3-395c-4e5a-b565-001d9b8cf7f6" containerID="25e2fe728004d5f187a614801b42224d9d87b9ad12dfda671bad5cc07f2986aa" exitCode=0 Oct 30 19:58:59.761943 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.761850 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" Oct 30 19:58:59.761943 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.761884 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" event={"ID":"a9420be3-395c-4e5a-b565-001d9b8cf7f6","Type":"ContainerDied","Data":"25e2fe728004d5f187a614801b42224d9d87b9ad12dfda671bad5cc07f2986aa"} Oct 30 19:58:59.762115 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.761957 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f" event={"ID":"a9420be3-395c-4e5a-b565-001d9b8cf7f6","Type":"ContainerDied","Data":"7dabe3bc7e971dadaf76f673c11ebe885606432c815cb4b3c9dfde77c3359d0a"} Oct 30 19:58:59.762115 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.762009 4433 scope.go:122] "RemoveContainer" containerID="25e2fe728004d5f187a614801b42224d9d87b9ad12dfda671bad5cc07f2986aa" Oct 30 19:58:59.762607 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.762581 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-podinfo" pod "a9420be3-395c-4e5a-b565-001d9b8cf7f6" (UID: "a9420be3-395c-4e5a-b565-001d9b8cf7f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:59.762712 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.762663 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9420be3-395c-4e5a-b565-001d9b8cf7f6-kube-api-access-rfgpk" pod "a9420be3-395c-4e5a-b565-001d9b8cf7f6" (UID: "a9420be3-395c-4e5a-b565-001d9b8cf7f6"). InnerVolumeSpecName "kube-api-access-rfgpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:59.762772 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.762717 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-envoy" pod "a9420be3-395c-4e5a-b565-001d9b8cf7f6" (UID: "a9420be3-395c-4e5a-b565-001d9b8cf7f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:59.763014 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.762961 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-token" pod "a9420be3-395c-4e5a-b565-001d9b8cf7f6" (UID: "a9420be3-395c-4e5a-b565-001d9b8cf7f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:59.787532 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.787507 4433 scope.go:122] "RemoveContainer" containerID="25e2fe728004d5f187a614801b42224d9d87b9ad12dfda671bad5cc07f2986aa" Oct 30 19:58:59.787776 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:58:59.787761 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e2fe728004d5f187a614801b42224d9d87b9ad12dfda671bad5cc07f2986aa\": container with ID starting with 25e2fe728004d5f187a614801b42224d9d87b9ad12dfda671bad5cc07f2986aa not found: ID does not exist" containerID="25e2fe728004d5f187a614801b42224d9d87b9ad12dfda671bad5cc07f2986aa" Oct 30 19:58:59.787827 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.787785 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e2fe728004d5f187a614801b42224d9d87b9ad12dfda671bad5cc07f2986aa"} err="failed to get container status \"25e2fe728004d5f187a614801b42224d9d87b9ad12dfda671bad5cc07f2986aa\": rpc error: code = NotFound desc = could not find container \"25e2fe728004d5f187a614801b42224d9d87b9ad12dfda671bad5cc07f2986aa\": container with ID starting with 25e2fe728004d5f187a614801b42224d9d87b9ad12dfda671bad5cc07f2986aa not found: ID does not exist" Oct 30 19:58:59.862434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.862404 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.862434 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.862432 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfgpk\" (UniqueName: \"kubernetes.io/projected/a9420be3-395c-4e5a-b565-001d9b8cf7f6-kube-api-access-rfgpk\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.862576 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.862452 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.862576 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.862460 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.862576 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:58:59.862468 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9420be3-395c-4e5a-b565-001d9b8cf7f6-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:00.026736 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:00.026654 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:00.212083 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:00.212032 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f"] Oct 30 19:59:00.217711 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:00.217672 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dff2f"] Oct 30 19:59:00.315207 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:00.315175 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:01.214823 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:01.214786 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9420be3-395c-4e5a-b565-001d9b8cf7f6" path="/var/lib/kubelet/pods/a9420be3-395c-4e5a-b565-001d9b8cf7f6/volumes" Oct 30 19:59:03.665066 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:03.665006 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" podUID="82dbd69a-c99e-4ea5-b2d3-471707b0d89c" containerName="payload-processing" containerID="cri-o://289a836b2bf8eb9ce170d040b6af42149e0b3d861fb54b632fe00b7e53453402" gracePeriod=25 Oct 30 19:59:03.788881 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:03.788856 4433 generic.go:356] "Generic (PLEG): container finished" podID="82dbd69a-c99e-4ea5-b2d3-471707b0d89c" containerID="289a836b2bf8eb9ce170d040b6af42149e0b3d861fb54b632fe00b7e53453402" exitCode=0 Oct 30 19:59:03.789007 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:03.788925 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" event={"ID":"82dbd69a-c99e-4ea5-b2d3-471707b0d89c","Type":"ContainerDied","Data":"289a836b2bf8eb9ce170d040b6af42149e0b3d861fb54b632fe00b7e53453402"} Oct 30 19:59:03.950879 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:03.950855 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" Oct 30 19:59:04.015561 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:04.015534 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82dbd69a-c99e-4ea5-b2d3-471707b0d89c-kube-api-access-whw9b\" (UniqueName: \"kubernetes.io/projected/82dbd69a-c99e-4ea5-b2d3-471707b0d89c-kube-api-access-whw9b\") pod \"82dbd69a-c99e-4ea5-b2d3-471707b0d89c\" (UID: \"82dbd69a-c99e-4ea5-b2d3-471707b0d89c\") " Oct 30 19:59:04.015792 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:04.015771 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82dbd69a-c99e-4ea5-b2d3-471707b0d89c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82dbd69a-c99e-4ea5-b2d3-471707b0d89c-plugins-config-volume\") pod \"82dbd69a-c99e-4ea5-b2d3-471707b0d89c\" (UID: \"82dbd69a-c99e-4ea5-b2d3-471707b0d89c\") " Oct 30 19:59:04.016131 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:04.016107 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82dbd69a-c99e-4ea5-b2d3-471707b0d89c-plugins-config-volume" pod "82dbd69a-c99e-4ea5-b2d3-471707b0d89c" (UID: "82dbd69a-c99e-4ea5-b2d3-471707b0d89c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:04.016285 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:04.016269 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82dbd69a-c99e-4ea5-b2d3-471707b0d89c-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:04.017495 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:04.017473 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82dbd69a-c99e-4ea5-b2d3-471707b0d89c-kube-api-access-whw9b" pod "82dbd69a-c99e-4ea5-b2d3-471707b0d89c" (UID: "82dbd69a-c99e-4ea5-b2d3-471707b0d89c"). InnerVolumeSpecName "kube-api-access-whw9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:04.117242 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:04.117213 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whw9b\" (UniqueName: \"kubernetes.io/projected/82dbd69a-c99e-4ea5-b2d3-471707b0d89c-kube-api-access-whw9b\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:04.304166 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:04.304085 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:04.572295 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:04.572254 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:04.796052 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:04.796018 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" Oct 30 19:59:04.796456 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:04.796022 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6" event={"ID":"82dbd69a-c99e-4ea5-b2d3-471707b0d89c","Type":"ContainerDied","Data":"5ff7099872861ae760c53ec709dbd2f0cd68c8228ac09f93bf2b7a89b4bb6a4e"} Oct 30 19:59:04.796456 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:04.796151 4433 scope.go:122] "RemoveContainer" containerID="289a836b2bf8eb9ce170d040b6af42149e0b3d861fb54b632fe00b7e53453402" Oct 30 19:59:04.833423 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:04.831190 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:04.960967 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:04.960934 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6"] Oct 30 19:59:04.966163 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:04.966136 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75b38d-bdbf4746d-f9jz6"] Oct 30 19:59:05.214440 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:05.214368 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82dbd69a-c99e-4ea5-b2d3-471707b0d89c" path="/var/lib/kubelet/pods/82dbd69a-c99e-4ea5-b2d3-471707b0d89c/volumes" Oct 30 19:59:06.913214 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:06.913182 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:07.173834 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:07.173750 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:07.419246 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:07.419211 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:09.327431 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:09.327402 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:10.673470 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:10.662620 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:10.994144 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:10.993566 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:11.251219 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:11.251111 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:11.941752 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:11.941724 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:12.207740 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:12.207651 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:12.502456 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:12.502338 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:12.782780 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:12.782694 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:13.303398 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:13.303346 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:17.851608 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:17.851575 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:18.556884 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:18.556819 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:18.896280 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:18.896247 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx"] Oct 30 19:59:18.896713 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:18.896469 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" podUID="096a248d-1aed-493d-9ec1-99bf2a8217eb" containerName="istio-proxy" containerID="cri-o://55bf821183c265f7ff5a2c706f4136c40d23125f896d6cb102ee915ea7faf3aa" gracePeriod=30 Oct 30 19:59:19.047926 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:19.047871 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:19.382859 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:19.382824 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:21.429450 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:21.429414 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:21.826017 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:21.823254 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:22.096849 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:22.096768 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:23.313426 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:23.313396 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:23.366357 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:59:23.366326 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:59:24.158690 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.158639 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:59:24.179691 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.179665 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/096a248d-1aed-493d-9ec1-99bf2a8217eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/096a248d-1aed-493d-9ec1-99bf2a8217eb-istiod-ca-cert\") pod \"096a248d-1aed-493d-9ec1-99bf2a8217eb\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " Oct 30 19:59:24.179803 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.179730 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-workload-socket\") pod \"096a248d-1aed-493d-9ec1-99bf2a8217eb\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " Oct 30 19:59:24.179803 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.179782 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096a248d-1aed-493d-9ec1-99bf2a8217eb-kube-api-access-lrxtk\" (UniqueName: \"kubernetes.io/projected/096a248d-1aed-493d-9ec1-99bf2a8217eb-kube-api-access-lrxtk\") pod \"096a248d-1aed-493d-9ec1-99bf2a8217eb\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " Oct 30 19:59:24.179889 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.179806 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-podinfo\") pod \"096a248d-1aed-493d-9ec1-99bf2a8217eb\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " Oct 30 19:59:24.179889 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.179856 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-workload-certs\") pod \"096a248d-1aed-493d-9ec1-99bf2a8217eb\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " Oct 30 19:59:24.180013 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.179906 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-data\") pod \"096a248d-1aed-493d-9ec1-99bf2a8217eb\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " Oct 30 19:59:24.180013 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.180005 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-token\" (UniqueName: \"kubernetes.io/projected/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-token\") pod \"096a248d-1aed-493d-9ec1-99bf2a8217eb\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " Oct 30 19:59:24.180133 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.180039 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-credential-socket\") pod \"096a248d-1aed-493d-9ec1-99bf2a8217eb\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " Oct 30 19:59:24.180133 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.180041 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096a248d-1aed-493d-9ec1-99bf2a8217eb-istiod-ca-cert" pod "096a248d-1aed-493d-9ec1-99bf2a8217eb" (UID: "096a248d-1aed-493d-9ec1-99bf2a8217eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:24.180133 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.180073 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-workload-socket" pod "096a248d-1aed-493d-9ec1-99bf2a8217eb" (UID: "096a248d-1aed-493d-9ec1-99bf2a8217eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:24.180133 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.180081 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-envoy\") pod \"096a248d-1aed-493d-9ec1-99bf2a8217eb\" (UID: \"096a248d-1aed-493d-9ec1-99bf2a8217eb\") " Oct 30 19:59:24.180342 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.180148 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-workload-certs" pod "096a248d-1aed-493d-9ec1-99bf2a8217eb" (UID: "096a248d-1aed-493d-9ec1-99bf2a8217eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:24.180402 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.180303 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-data" pod "096a248d-1aed-493d-9ec1-99bf2a8217eb" (UID: "096a248d-1aed-493d-9ec1-99bf2a8217eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:24.180455 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.180380 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-credential-socket" pod "096a248d-1aed-493d-9ec1-99bf2a8217eb" (UID: "096a248d-1aed-493d-9ec1-99bf2a8217eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:24.180876 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.180849 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/096a248d-1aed-493d-9ec1-99bf2a8217eb-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:24.181019 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.180880 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:24.181019 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.180895 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:24.181019 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.180909 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:24.181019 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.180923 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:24.182016 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.181967 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-podinfo" pod "096a248d-1aed-493d-9ec1-99bf2a8217eb" (UID: "096a248d-1aed-493d-9ec1-99bf2a8217eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:59:24.182258 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.182233 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-token" pod "096a248d-1aed-493d-9ec1-99bf2a8217eb" (UID: "096a248d-1aed-493d-9ec1-99bf2a8217eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:24.182373 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.182354 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-envoy" pod "096a248d-1aed-493d-9ec1-99bf2a8217eb" (UID: "096a248d-1aed-493d-9ec1-99bf2a8217eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:24.182423 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.182393 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096a248d-1aed-493d-9ec1-99bf2a8217eb-kube-api-access-lrxtk" pod "096a248d-1aed-493d-9ec1-99bf2a8217eb" (UID: "096a248d-1aed-493d-9ec1-99bf2a8217eb"). InnerVolumeSpecName "kube-api-access-lrxtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:24.282635 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.282612 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrxtk\" (UniqueName: \"kubernetes.io/projected/096a248d-1aed-493d-9ec1-99bf2a8217eb-kube-api-access-lrxtk\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:24.282768 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.282638 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:24.282768 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.282652 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:24.282768 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.282665 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/096a248d-1aed-493d-9ec1-99bf2a8217eb-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:24.978609 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.978570 4433 generic.go:356] "Generic (PLEG): container finished" podID="096a248d-1aed-493d-9ec1-99bf2a8217eb" containerID="55bf821183c265f7ff5a2c706f4136c40d23125f896d6cb102ee915ea7faf3aa" exitCode=0 Oct 30 19:59:24.979061 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.978627 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" Oct 30 19:59:24.979061 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.978658 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" event={"ID":"096a248d-1aed-493d-9ec1-99bf2a8217eb","Type":"ContainerDied","Data":"55bf821183c265f7ff5a2c706f4136c40d23125f896d6cb102ee915ea7faf3aa"} Oct 30 19:59:24.979061 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.978715 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx" event={"ID":"096a248d-1aed-493d-9ec1-99bf2a8217eb","Type":"ContainerDied","Data":"e91876d37f81a0cbc28d7174d96c034846faf3357766897f35bf97b13b2b9d9c"} Oct 30 19:59:24.979061 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.978737 4433 scope.go:122] "RemoveContainer" containerID="55bf821183c265f7ff5a2c706f4136c40d23125f896d6cb102ee915ea7faf3aa" Oct 30 19:59:24.989522 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.989493 4433 scope.go:122] "RemoveContainer" containerID="6e66a979d6dd47eaf68da344e7f2c4e5c4cf5f5629edbe513d4a3451ecc01ed9" Oct 30 19:59:24.998963 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.998944 4433 scope.go:122] "RemoveContainer" containerID="55bf821183c265f7ff5a2c706f4136c40d23125f896d6cb102ee915ea7faf3aa" Oct 30 19:59:24.999301 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:59:24.999280 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55bf821183c265f7ff5a2c706f4136c40d23125f896d6cb102ee915ea7faf3aa\": container with ID starting with 55bf821183c265f7ff5a2c706f4136c40d23125f896d6cb102ee915ea7faf3aa not found: ID does not exist" containerID="55bf821183c265f7ff5a2c706f4136c40d23125f896d6cb102ee915ea7faf3aa" Oct 30 19:59:24.999345 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.999313 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55bf821183c265f7ff5a2c706f4136c40d23125f896d6cb102ee915ea7faf3aa"} err="failed to get container status \"55bf821183c265f7ff5a2c706f4136c40d23125f896d6cb102ee915ea7faf3aa\": rpc error: code = NotFound desc = could not find container \"55bf821183c265f7ff5a2c706f4136c40d23125f896d6cb102ee915ea7faf3aa\": container with ID starting with 55bf821183c265f7ff5a2c706f4136c40d23125f896d6cb102ee915ea7faf3aa not found: ID does not exist" Oct 30 19:59:24.999345 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.999332 4433 scope.go:122] "RemoveContainer" containerID="6e66a979d6dd47eaf68da344e7f2c4e5c4cf5f5629edbe513d4a3451ecc01ed9" Oct 30 19:59:24.999623 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:59:24.999602 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e66a979d6dd47eaf68da344e7f2c4e5c4cf5f5629edbe513d4a3451ecc01ed9\": container with ID starting with 6e66a979d6dd47eaf68da344e7f2c4e5c4cf5f5629edbe513d4a3451ecc01ed9 not found: ID does not exist" containerID="6e66a979d6dd47eaf68da344e7f2c4e5c4cf5f5629edbe513d4a3451ecc01ed9" Oct 30 19:59:24.999672 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:24.999630 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e66a979d6dd47eaf68da344e7f2c4e5c4cf5f5629edbe513d4a3451ecc01ed9"} err="failed to get container status \"6e66a979d6dd47eaf68da344e7f2c4e5c4cf5f5629edbe513d4a3451ecc01ed9\": rpc error: code = NotFound desc = could not find container \"6e66a979d6dd47eaf68da344e7f2c4e5c4cf5f5629edbe513d4a3451ecc01ed9\": container with ID starting with 6e66a979d6dd47eaf68da344e7f2c4e5c4cf5f5629edbe513d4a3451ecc01ed9 not found: ID does not exist" Oct 30 19:59:25.194865 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:25.194821 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx"] Oct 30 19:59:25.244958 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:25.244879 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-75b38d-openshift-default-d58bd8ff-xr8hx"] Oct 30 19:59:25.960419 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:25.960386 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:27.214819 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:27.214789 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="096a248d-1aed-493d-9ec1-99bf2a8217eb" path="/var/lib/kubelet/pods/096a248d-1aed-493d-9ec1-99bf2a8217eb/volumes" Oct 30 19:59:28.471273 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:28.471239 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:31.756385 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:31.756343 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:32.002348 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:32.002308 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:33.410647 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:33.410604 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:34.414521 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:34.414485 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:34.668975 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:34.668872 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:38.106481 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:38.106443 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:40.557873 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:40.557843 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:40.795430 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:40.795383 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:41.996528 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:41.996490 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv"] Oct 30 19:59:41.996967 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:41.996731 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" podUID="e1409587-03bd-41d3-93cb-83d38bd9a4f2" containerName="maas-api" containerID="cri-o://6ffc52b3706d44e2f6e809c62ffc803fb440175bbfb3033ba76aa1c0fde4c186" gracePeriod=30 Oct 30 19:59:42.194018 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:42.193639 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:42.318573 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:42.318544 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:59:42.447852 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:42.447814 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1409587-03bd-41d3-93cb-83d38bd9a4f2-kube-api-access-xqbqv\" (UniqueName: \"kubernetes.io/projected/e1409587-03bd-41d3-93cb-83d38bd9a4f2-kube-api-access-xqbqv\") pod \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " Oct 30 19:59:42.448042 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:42.447900 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-metrics-certs\") pod \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " Oct 30 19:59:42.448121 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:42.448088 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-maas-api-tls\") pod \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " Oct 30 19:59:42.448258 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:42.448230 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1409587-03bd-41d3-93cb-83d38bd9a4f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1409587-03bd-41d3-93cb-83d38bd9a4f2-openshift-service-ca\") pod \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\" (UID: \"e1409587-03bd-41d3-93cb-83d38bd9a4f2\") " Oct 30 19:59:42.448849 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:42.448819 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1409587-03bd-41d3-93cb-83d38bd9a4f2-openshift-service-ca" pod "e1409587-03bd-41d3-93cb-83d38bd9a4f2" (UID: "e1409587-03bd-41d3-93cb-83d38bd9a4f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:42.450799 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:42.450771 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1409587-03bd-41d3-93cb-83d38bd9a4f2-kube-api-access-xqbqv" pod "e1409587-03bd-41d3-93cb-83d38bd9a4f2" (UID: "e1409587-03bd-41d3-93cb-83d38bd9a4f2"). InnerVolumeSpecName "kube-api-access-xqbqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:42.462135 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:42.462106 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-maas-api-tls" pod "e1409587-03bd-41d3-93cb-83d38bd9a4f2" (UID: "e1409587-03bd-41d3-93cb-83d38bd9a4f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:42.468114 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:42.468086 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-metrics-certs" pod "e1409587-03bd-41d3-93cb-83d38bd9a4f2" (UID: "e1409587-03bd-41d3-93cb-83d38bd9a4f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:42.549959 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:42.549908 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1409587-03bd-41d3-93cb-83d38bd9a4f2-openshift-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:42.549959 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:42.549929 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqbqv\" (UniqueName: \"kubernetes.io/projected/e1409587-03bd-41d3-93cb-83d38bd9a4f2-kube-api-access-xqbqv\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:42.549959 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:42.549941 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-metrics-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:42.549959 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:42.549949 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1409587-03bd-41d3-93cb-83d38bd9a4f2-maas-api-tls\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:43.114914 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:43.114877 4433 generic.go:356] "Generic (PLEG): container finished" podID="e1409587-03bd-41d3-93cb-83d38bd9a4f2" containerID="6ffc52b3706d44e2f6e809c62ffc803fb440175bbfb3033ba76aa1c0fde4c186" exitCode=0 Oct 30 19:59:43.115344 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:43.114926 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" Oct 30 19:59:43.115344 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:43.114938 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" event={"ID":"e1409587-03bd-41d3-93cb-83d38bd9a4f2","Type":"ContainerDied","Data":"6ffc52b3706d44e2f6e809c62ffc803fb440175bbfb3033ba76aa1c0fde4c186"} Oct 30 19:59:43.115344 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:43.115040 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv" event={"ID":"e1409587-03bd-41d3-93cb-83d38bd9a4f2","Type":"ContainerDied","Data":"30a8f8b765a48e5b6efac2122efef396edd8903763d63b19a4542a011c1b8b1e"} Oct 30 19:59:43.115344 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:43.115070 4433 scope.go:122] "RemoveContainer" containerID="6ffc52b3706d44e2f6e809c62ffc803fb440175bbfb3033ba76aa1c0fde4c186" Oct 30 19:59:43.125399 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:43.125377 4433 scope.go:122] "RemoveContainer" containerID="6ffc52b3706d44e2f6e809c62ffc803fb440175bbfb3033ba76aa1c0fde4c186" Oct 30 19:59:43.125651 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:59:43.125627 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ffc52b3706d44e2f6e809c62ffc803fb440175bbfb3033ba76aa1c0fde4c186\": container with ID starting with 6ffc52b3706d44e2f6e809c62ffc803fb440175bbfb3033ba76aa1c0fde4c186 not found: ID does not exist" containerID="6ffc52b3706d44e2f6e809c62ffc803fb440175bbfb3033ba76aa1c0fde4c186" Oct 30 19:59:43.125709 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:43.125660 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ffc52b3706d44e2f6e809c62ffc803fb440175bbfb3033ba76aa1c0fde4c186"} err="failed to get container status \"6ffc52b3706d44e2f6e809c62ffc803fb440175bbfb3033ba76aa1c0fde4c186\": rpc error: code = NotFound desc = could not find container \"6ffc52b3706d44e2f6e809c62ffc803fb440175bbfb3033ba76aa1c0fde4c186\": container with ID starting with 6ffc52b3706d44e2f6e809c62ffc803fb440175bbfb3033ba76aa1c0fde4c186 not found: ID does not exist" Oct 30 19:59:43.290574 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:43.290544 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv"] Oct 30 19:59:43.299448 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:43.299411 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da4c62-5499dd465-ld8tv"] Oct 30 19:59:43.764115 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:43.764082 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:44.042332 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:44.042246 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:44.322816 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:44.322784 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:44.746943 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:44.746857 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:45.215696 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:45.215665 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1409587-03bd-41d3-93cb-83d38bd9a4f2" path="/var/lib/kubelet/pods/e1409587-03bd-41d3-93cb-83d38bd9a4f2/volumes" Oct 30 19:59:48.707354 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:48.707288 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:49.663599 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:49.663544 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:49.911201 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:49.911090 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:50.224564 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:50.224530 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:50.730016 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:50.729185 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:51.392456 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:51.392426 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:51.764524 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:51.764446 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:52.005724 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:52.005689 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:52.437439 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:52.437406 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:52.832420 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:52.832384 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:53.366719 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:59:53.366682 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 19:59:56.357620 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.357585 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw"] Oct 30 19:59:56.358113 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.357788 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" podUID="48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9" containerName="maas-api" containerID="cri-o://09568bd15df887959b0afc56a5e07b7b10a4ef36d8d32abae9afe1a11e54a7e6" gracePeriod=30 Oct 30 19:59:56.380543 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.380509 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" podUID="48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.55:8443/health\": dial tcp 10.134.0.55:8443: connect: connection refused" Oct 30 19:59:56.564609 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.564574 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:56.648298 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.648268 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:59:56.713309 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.713258 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-metrics-certs\") pod \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " Oct 30 19:59:56.713491 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.713345 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-openshift-service-ca\") pod \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " Oct 30 19:59:56.713491 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.713404 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-maas-api-tls\") pod \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " Oct 30 19:59:56.713491 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.713460 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-kube-api-access-84bn4\" (UniqueName: \"kubernetes.io/projected/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-kube-api-access-84bn4\") pod \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\" (UID: \"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9\") " Oct 30 19:59:56.713738 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.713709 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-openshift-service-ca" pod "48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9" (UID: "48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:56.713877 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.713863 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-openshift-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:56.715568 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.715545 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-metrics-certs" pod "48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9" (UID: "48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:56.715568 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.715553 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-kube-api-access-84bn4" pod "48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9" (UID: "48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9"). InnerVolumeSpecName "kube-api-access-84bn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:56.715697 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.715570 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-maas-api-tls" pod "48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9" (UID: "48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:56.815028 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.814988 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-metrics-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:56.815028 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.815024 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-maas-api-tls\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:56.815028 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:56.815034 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84bn4\" (UniqueName: \"kubernetes.io/projected/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9-kube-api-access-84bn4\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:59:57.204879 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:57.204841 4433 generic.go:356] "Generic (PLEG): container finished" podID="48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9" containerID="09568bd15df887959b0afc56a5e07b7b10a4ef36d8d32abae9afe1a11e54a7e6" exitCode=0 Oct 30 19:59:57.205122 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:57.204912 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" event={"ID":"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9","Type":"ContainerDied","Data":"09568bd15df887959b0afc56a5e07b7b10a4ef36d8d32abae9afe1a11e54a7e6"} Oct 30 19:59:57.205122 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:57.204968 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" Oct 30 19:59:57.205122 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:57.205008 4433 scope.go:122] "RemoveContainer" containerID="09568bd15df887959b0afc56a5e07b7b10a4ef36d8d32abae9afe1a11e54a7e6" Oct 30 19:59:57.205122 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:57.204993 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw" event={"ID":"48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9","Type":"ContainerDied","Data":"bc550908ecb1741d0df4d1c2bacdfbd12a9918e186f09dd22216631e8ca15429"} Oct 30 19:59:57.217776 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:57.217719 4433 scope.go:122] "RemoveContainer" containerID="09568bd15df887959b0afc56a5e07b7b10a4ef36d8d32abae9afe1a11e54a7e6" Oct 30 19:59:57.218527 ip-10-0-141-5 kubenswrapper[4433]: E1030 19:59:57.218208 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09568bd15df887959b0afc56a5e07b7b10a4ef36d8d32abae9afe1a11e54a7e6\": container with ID starting with 09568bd15df887959b0afc56a5e07b7b10a4ef36d8d32abae9afe1a11e54a7e6 not found: ID does not exist" containerID="09568bd15df887959b0afc56a5e07b7b10a4ef36d8d32abae9afe1a11e54a7e6" Oct 30 19:59:57.218527 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:57.218246 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09568bd15df887959b0afc56a5e07b7b10a4ef36d8d32abae9afe1a11e54a7e6"} err="failed to get container status \"09568bd15df887959b0afc56a5e07b7b10a4ef36d8d32abae9afe1a11e54a7e6\": rpc error: code = NotFound desc = could not find container \"09568bd15df887959b0afc56a5e07b7b10a4ef36d8d32abae9afe1a11e54a7e6\": container with ID starting with 09568bd15df887959b0afc56a5e07b7b10a4ef36d8d32abae9afe1a11e54a7e6 not found: ID does not exist" Oct 30 19:59:57.300415 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:57.300382 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j"] Oct 30 19:59:57.377015 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:57.376967 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw"] Oct 30 19:59:57.389085 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:57.389052 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d52df0-c5fc5cbd6-hpzkw"] Oct 30 19:59:57.597414 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:57.597385 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:57.905475 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:57.905388 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:58.161232 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:58.161125 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:58.464418 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:58.464336 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:58.833402 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:58.833365 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:59.084649 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:59.084546 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:59.087817 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:59.087789 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd"] Oct 30 19:59:59.088079 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:59.088056 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" podUID="f1e0904a-f62b-4828-b130-fdef755b3831" containerName="istio-proxy" containerID="cri-o://b7565739b966ce2722d086cd53855837ae23c0b7ee2258b0125eb27961369afb" gracePeriod=30 Oct 30 19:59:59.236437 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:59.236396 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9" path="/var/lib/kubelet/pods/48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9/volumes" Oct 30 19:59:59.333408 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:59.333366 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:59.584901 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:59.584864 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:59.869055 ip-10-0-141-5 kubenswrapper[4433]: I1030 19:59:59.868904 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:00.194230 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194144 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d"] Oct 30 20:00:00.194711 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194699 4433 cpu_manager.go:414] "containerMap: removing container" podUID="82dbd69a-c99e-4ea5-b2d3-471707b0d89c" containerName="payload-processing" Oct 30 20:00:00.194758 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194712 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82dbd69a-c99e-4ea5-b2d3-471707b0d89c" containerName="payload-processing" Oct 30 20:00:00.194758 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194726 4433 cpu_manager.go:414] "containerMap: removing container" podUID="096a248d-1aed-493d-9ec1-99bf2a8217eb" containerName="istio-proxy" Oct 30 20:00:00.194758 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194732 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096a248d-1aed-493d-9ec1-99bf2a8217eb" containerName="istio-proxy" Oct 30 20:00:00.194758 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194745 4433 cpu_manager.go:414] "containerMap: removing container" podUID="096a248d-1aed-493d-9ec1-99bf2a8217eb" containerName="istio-proxy" Oct 30 20:00:00.194758 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194750 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096a248d-1aed-493d-9ec1-99bf2a8217eb" containerName="istio-proxy" Oct 30 20:00:00.194758 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194758 4433 cpu_manager.go:414] "containerMap: removing container" podUID="e1409587-03bd-41d3-93cb-83d38bd9a4f2" containerName="maas-api" Oct 30 20:00:00.194935 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194763 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1409587-03bd-41d3-93cb-83d38bd9a4f2" containerName="maas-api" Oct 30 20:00:00.194935 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194770 4433 cpu_manager.go:414] "containerMap: removing container" podUID="5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" containerName="istio-proxy" Oct 30 20:00:00.194935 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194777 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" containerName="istio-proxy" Oct 30 20:00:00.194935 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194785 4433 cpu_manager.go:414] "containerMap: removing container" podUID="a9420be3-395c-4e5a-b565-001d9b8cf7f6" containerName="istio-proxy" Oct 30 20:00:00.194935 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194789 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9420be3-395c-4e5a-b565-001d9b8cf7f6" containerName="istio-proxy" Oct 30 20:00:00.194935 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194799 4433 cpu_manager.go:414] "containerMap: removing container" podUID="4026634f-e0b5-4b64-a0ce-ebfe65c07365" containerName="payload-pre-processing" Oct 30 20:00:00.194935 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194803 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4026634f-e0b5-4b64-a0ce-ebfe65c07365" containerName="payload-pre-processing" Oct 30 20:00:00.194935 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194814 4433 cpu_manager.go:414] "containerMap: removing container" podUID="48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9" containerName="maas-api" Oct 30 20:00:00.194935 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194819 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9" containerName="maas-api" Oct 30 20:00:00.194935 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194830 4433 cpu_manager.go:414] "containerMap: removing container" podUID="666f2ff7-bd9c-4525-bdf9-bda82db33b9a" containerName="istio-proxy" Oct 30 20:00:00.194935 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194834 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666f2ff7-bd9c-4525-bdf9-bda82db33b9a" containerName="istio-proxy" Oct 30 20:00:00.194935 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194850 4433 cpu_manager.go:414] "containerMap: removing container" podUID="bb823cde-6a63-4fcd-acc7-4700d65ba7ac" containerName="payload-processing" Oct 30 20:00:00.194935 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194855 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb823cde-6a63-4fcd-acc7-4700d65ba7ac" containerName="payload-processing" Oct 30 20:00:00.194935 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194934 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a83cf5-b6bb-4b67-ab3c-d4c0c42c9fc9" containerName="maas-api" Oct 30 20:00:00.195364 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194942 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="666f2ff7-bd9c-4525-bdf9-bda82db33b9a" containerName="istio-proxy" Oct 30 20:00:00.195364 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194951 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb823cde-6a63-4fcd-acc7-4700d65ba7ac" containerName="payload-processing" Oct 30 20:00:00.195364 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194959 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="82dbd69a-c99e-4ea5-b2d3-471707b0d89c" containerName="payload-processing" Oct 30 20:00:00.195364 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194966 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="096a248d-1aed-493d-9ec1-99bf2a8217eb" containerName="istio-proxy" Oct 30 20:00:00.195364 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194972 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1409587-03bd-41d3-93cb-83d38bd9a4f2" containerName="maas-api" Oct 30 20:00:00.195364 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.194993 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9ef6f1-ed94-474c-9da2-d2cc68f3ec8a" containerName="istio-proxy" Oct 30 20:00:00.195364 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.195000 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9420be3-395c-4e5a-b565-001d9b8cf7f6" containerName="istio-proxy" Oct 30 20:00:00.195364 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.195008 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4026634f-e0b5-4b64-a0ce-ebfe65c07365" containerName="payload-pre-processing" Oct 30 20:00:00.199788 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.199769 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" Oct 30 20:00:00.206353 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.206330 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6"] Oct 30 20:00:00.207185 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.207162 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="096a248d-1aed-493d-9ec1-99bf2a8217eb" containerName="istio-proxy" Oct 30 20:00:00.210802 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.210781 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" Oct 30 20:00:00.211192 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.211168 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn"] Oct 30 20:00:00.215399 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.215374 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" Oct 30 20:00:00.226019 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.225965 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn"] Oct 30 20:00:00.230105 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.230086 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" Oct 30 20:00:00.234813 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.234781 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:00.281013 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.280963 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d"] Oct 30 20:00:00.284464 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.284436 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56448e26-174f-4edd-96a7-87cbd714ad8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn\" (UID: \"56448e26-174f-4edd-96a7-87cbd714ad8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" Oct 30 20:00:00.284627 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.284494 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg77r\" (UniqueName: \"kubernetes.io/projected/56448e26-174f-4edd-96a7-87cbd714ad8e-kube-api-access-kg77r\") pod \"maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn\" (UID: \"56448e26-174f-4edd-96a7-87cbd714ad8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" Oct 30 20:00:00.284747 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.284721 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5nsb\" (UniqueName: \"kubernetes.io/projected/2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac-kube-api-access-q5nsb\") pod \"maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d\" (UID: \"2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" Oct 30 20:00:00.284903 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.284883 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d\" (UID: \"2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" Oct 30 20:00:00.285022 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.284928 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded326eb-7db6-432f-8427-68833723a5fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6\" (UID: \"ded326eb-7db6-432f-8427-68833723a5fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" Oct 30 20:00:00.285022 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.284954 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98bl9\" (UniqueName: \"kubernetes.io/projected/ded326eb-7db6-432f-8427-68833723a5fa-kube-api-access-98bl9\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6\" (UID: \"ded326eb-7db6-432f-8427-68833723a5fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" Oct 30 20:00:00.304473 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.304444 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn"] Oct 30 20:00:00.323423 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.323391 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn"] Oct 30 20:00:00.337616 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.337583 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\"" Oct 30 20:00:00.357533 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.357501 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6"] Oct 30 20:00:00.386663 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.386618 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37be5208-015f-4ab4-8bd0-f390d3729293-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-ffddn\" (UID: \"37be5208-015f-4ab4-8bd0-f390d3729293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" Oct 30 20:00:00.386803 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.386690 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d\" (UID: \"2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" Oct 30 20:00:00.386803 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.386715 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded326eb-7db6-432f-8427-68833723a5fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6\" (UID: \"ded326eb-7db6-432f-8427-68833723a5fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" Oct 30 20:00:00.386803 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.386730 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bl9\" (UniqueName: \"kubernetes.io/projected/ded326eb-7db6-432f-8427-68833723a5fa-kube-api-access-98bl9\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6\" (UID: \"ded326eb-7db6-432f-8427-68833723a5fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" Oct 30 20:00:00.386803 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.386785 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56448e26-174f-4edd-96a7-87cbd714ad8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn\" (UID: \"56448e26-174f-4edd-96a7-87cbd714ad8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" Oct 30 20:00:00.387004 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.386829 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg77r\" (UniqueName: \"kubernetes.io/projected/56448e26-174f-4edd-96a7-87cbd714ad8e-kube-api-access-kg77r\") pod \"maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn\" (UID: \"56448e26-174f-4edd-96a7-87cbd714ad8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" Oct 30 20:00:00.387004 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.386936 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nsb\" (UniqueName: \"kubernetes.io/projected/2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac-kube-api-access-q5nsb\") pod \"maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d\" (UID: \"2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" Oct 30 20:00:00.387125 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.387012 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbvhc\" (UniqueName: \"kubernetes.io/projected/37be5208-015f-4ab4-8bd0-f390d3729293-kube-api-access-sbvhc\") pod \"maas-api-key-cleanup-29889840-ffddn\" (UID: \"37be5208-015f-4ab4-8bd0-f390d3729293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" Oct 30 20:00:00.387470 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.387452 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56448e26-174f-4edd-96a7-87cbd714ad8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn\" (UID: \"56448e26-174f-4edd-96a7-87cbd714ad8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" Oct 30 20:00:00.387518 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.387468 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded326eb-7db6-432f-8427-68833723a5fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6\" (UID: \"ded326eb-7db6-432f-8427-68833723a5fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" Oct 30 20:00:00.387558 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.387540 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d\" (UID: \"2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" Oct 30 20:00:00.446660 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.446555 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98bl9\" (UniqueName: \"kubernetes.io/projected/ded326eb-7db6-432f-8427-68833723a5fa-kube-api-access-98bl9\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6\" (UID: \"ded326eb-7db6-432f-8427-68833723a5fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" Oct 30 20:00:00.446660 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.446617 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg77r\" (UniqueName: \"kubernetes.io/projected/56448e26-174f-4edd-96a7-87cbd714ad8e-kube-api-access-kg77r\") pod \"maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn\" (UID: \"56448e26-174f-4edd-96a7-87cbd714ad8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" Oct 30 20:00:00.446882 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.446703 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5nsb\" (UniqueName: \"kubernetes.io/projected/2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac-kube-api-access-q5nsb\") pod \"maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d\" (UID: \"2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" Oct 30 20:00:00.481893 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.481859 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:00.487789 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.487763 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbvhc\" (UniqueName: \"kubernetes.io/projected/37be5208-015f-4ab4-8bd0-f390d3729293-kube-api-access-sbvhc\") pod \"maas-api-key-cleanup-29889840-ffddn\" (UID: \"37be5208-015f-4ab4-8bd0-f390d3729293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" Oct 30 20:00:00.487929 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.487810 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37be5208-015f-4ab4-8bd0-f390d3729293-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-ffddn\" (UID: \"37be5208-015f-4ab4-8bd0-f390d3729293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" Oct 30 20:00:00.488442 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.488421 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37be5208-015f-4ab4-8bd0-f390d3729293-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-ffddn\" (UID: \"37be5208-015f-4ab4-8bd0-f390d3729293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" Oct 30 20:00:00.517450 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.517417 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" Oct 30 20:00:00.546093 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.529614 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" Oct 30 20:00:00.617900 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.617651 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" Oct 30 20:00:00.652080 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.651573 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbvhc\" (UniqueName: \"kubernetes.io/projected/37be5208-015f-4ab4-8bd0-f390d3729293-kube-api-access-sbvhc\") pod \"maas-api-key-cleanup-29889840-ffddn\" (UID: \"37be5208-015f-4ab4-8bd0-f390d3729293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" Oct 30 20:00:00.739645 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.738725 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:00.845661 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:00.845629 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" Oct 30 20:00:01.060659 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:01.060257 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6"] Oct 30 20:00:01.062772 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:00:01.062742 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb12b8c_9c22_4c09_8a9a_7637e3ce74ac.slice/crio-afec48fb6dd4246f211e7504af64771dfa2696da5558d7385f75687edd4b1aca WatchSource:0}: Error finding container afec48fb6dd4246f211e7504af64771dfa2696da5558d7385f75687edd4b1aca: Status 404 returned error can't find the container with id afec48fb6dd4246f211e7504af64771dfa2696da5558d7385f75687edd4b1aca Oct 30 20:00:01.063067 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:00:01.063041 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded326eb_7db6_432f_8427_68833723a5fa.slice/crio-8f91a55ba0b62ba7ab0a181a7191d9713e94a7aee3b56ce83f543ca446157d42 WatchSource:0}: Error finding container 8f91a55ba0b62ba7ab0a181a7191d9713e94a7aee3b56ce83f543ca446157d42: Status 404 returned error can't find the container with id 8f91a55ba0b62ba7ab0a181a7191d9713e94a7aee3b56ce83f543ca446157d42 Oct 30 20:00:01.065016 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:01.064992 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d"] Oct 30 20:00:01.108436 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:01.108410 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn"] Oct 30 20:00:01.110596 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:00:01.110570 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56448e26_174f_4edd_96a7_87cbd714ad8e.slice/crio-3685f727fd19ff04dc2761b1f53ff43c2ce024760371eeb19ebccd751407c672 WatchSource:0}: Error finding container 3685f727fd19ff04dc2761b1f53ff43c2ce024760371eeb19ebccd751407c672: Status 404 returned error can't find the container with id 3685f727fd19ff04dc2761b1f53ff43c2ce024760371eeb19ebccd751407c672 Oct 30 20:00:01.237013 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:01.236971 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn"] Oct 30 20:00:01.238675 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:00:01.238649 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37be5208_015f_4ab4_8bd0_f390d3729293.slice/crio-d1ccbc182c7e3f799598346dc1429b3c871491127e2ba6e9588ef8f066571886 WatchSource:0}: Error finding container d1ccbc182c7e3f799598346dc1429b3c871491127e2ba6e9588ef8f066571886: Status 404 returned error can't find the container with id d1ccbc182c7e3f799598346dc1429b3c871491127e2ba6e9588ef8f066571886 Oct 30 20:00:01.274463 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:01.274434 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" event={"ID":"2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac","Type":"ContainerStarted","Data":"afec48fb6dd4246f211e7504af64771dfa2696da5558d7385f75687edd4b1aca"} Oct 30 20:00:01.275707 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:01.275683 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" event={"ID":"56448e26-174f-4edd-96a7-87cbd714ad8e","Type":"ContainerStarted","Data":"3685f727fd19ff04dc2761b1f53ff43c2ce024760371eeb19ebccd751407c672"} Oct 30 20:00:01.276799 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:01.276777 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" event={"ID":"37be5208-015f-4ab4-8bd0-f390d3729293","Type":"ContainerStarted","Data":"d1ccbc182c7e3f799598346dc1429b3c871491127e2ba6e9588ef8f066571886"} Oct 30 20:00:01.277826 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:01.277806 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" event={"ID":"ded326eb-7db6-432f-8427-68833723a5fa","Type":"ContainerStarted","Data":"8f91a55ba0b62ba7ab0a181a7191d9713e94a7aee3b56ce83f543ca446157d42"} Oct 30 20:00:02.288045 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:02.287958 4433 generic.go:356] "Generic (PLEG): container finished" podID="2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" containerID="e51812bc819389148641e33a08f5a7bc4be765d8e207a24dcc9851002d8a5f3b" exitCode=60 Oct 30 20:00:02.288045 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:02.288031 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" event={"ID":"2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac","Type":"ContainerDied","Data":"e51812bc819389148641e33a08f5a7bc4be765d8e207a24dcc9851002d8a5f3b"} Oct 30 20:00:02.288481 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:02.288305 4433 scope.go:122] "RemoveContainer" containerID="e51812bc819389148641e33a08f5a7bc4be765d8e207a24dcc9851002d8a5f3b" Oct 30 20:00:02.289924 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:02.289805 4433 generic.go:356] "Generic (PLEG): container finished" podID="56448e26-174f-4edd-96a7-87cbd714ad8e" containerID="d0c16d4873b1d58c3b6af8eea1c29d93a0ef4ffa79f31f6dba773c9b41a50ed5" exitCode=60 Oct 30 20:00:02.289924 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:02.289837 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" event={"ID":"56448e26-174f-4edd-96a7-87cbd714ad8e","Type":"ContainerDied","Data":"d0c16d4873b1d58c3b6af8eea1c29d93a0ef4ffa79f31f6dba773c9b41a50ed5"} Oct 30 20:00:02.290134 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:02.290117 4433 scope.go:122] "RemoveContainer" containerID="d0c16d4873b1d58c3b6af8eea1c29d93a0ef4ffa79f31f6dba773c9b41a50ed5" Oct 30 20:00:02.291786 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:02.291632 4433 generic.go:356] "Generic (PLEG): container finished" podID="37be5208-015f-4ab4-8bd0-f390d3729293" containerID="b1ee6e7e9672968627bb4e4bc6367e38e44ab8d104a7f5ec37076d0a1baa1b13" exitCode=60 Oct 30 20:00:02.291786 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:02.291725 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" event={"ID":"37be5208-015f-4ab4-8bd0-f390d3729293","Type":"ContainerDied","Data":"b1ee6e7e9672968627bb4e4bc6367e38e44ab8d104a7f5ec37076d0a1baa1b13"} Oct 30 20:00:02.291942 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:02.291920 4433 scope.go:122] "RemoveContainer" containerID="b1ee6e7e9672968627bb4e4bc6367e38e44ab8d104a7f5ec37076d0a1baa1b13" Oct 30 20:00:02.293669 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:02.293646 4433 generic.go:356] "Generic (PLEG): container finished" podID="ded326eb-7db6-432f-8427-68833723a5fa" containerID="fc00e6c3dc35b3bb27d31fec87ec12fbb34df0c63004d678ac9d006758823905" exitCode=60 Oct 30 20:00:02.293763 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:02.293730 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" event={"ID":"ded326eb-7db6-432f-8427-68833723a5fa","Type":"ContainerDied","Data":"fc00e6c3dc35b3bb27d31fec87ec12fbb34df0c63004d678ac9d006758823905"} Oct 30 20:00:02.293946 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:02.293930 4433 scope.go:122] "RemoveContainer" containerID="fc00e6c3dc35b3bb27d31fec87ec12fbb34df0c63004d678ac9d006758823905" Oct 30 20:00:03.300854 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:03.300824 4433 generic.go:356] "Generic (PLEG): container finished" podID="56448e26-174f-4edd-96a7-87cbd714ad8e" containerID="f804b6c9c8afe3df8f1908f76ff8a542f6f58f5a48494290e678aa3fda7d5282" exitCode=60 Oct 30 20:00:03.301273 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:03.300884 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" event={"ID":"56448e26-174f-4edd-96a7-87cbd714ad8e","Type":"ContainerDied","Data":"f804b6c9c8afe3df8f1908f76ff8a542f6f58f5a48494290e678aa3fda7d5282"} Oct 30 20:00:03.301273 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:03.300959 4433 scope.go:122] "RemoveContainer" containerID="d0c16d4873b1d58c3b6af8eea1c29d93a0ef4ffa79f31f6dba773c9b41a50ed5" Oct 30 20:00:03.301273 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:03.301181 4433 scope.go:122] "RemoveContainer" containerID="f804b6c9c8afe3df8f1908f76ff8a542f6f58f5a48494290e678aa3fda7d5282" Oct 30 20:00:03.301422 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:03.301398 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn_odh-ai-gateway-infra(56448e26-174f-4edd-96a7-87cbd714ad8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" podUID="56448e26-174f-4edd-96a7-87cbd714ad8e" Oct 30 20:00:03.303043 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:03.303021 4433 generic.go:356] "Generic (PLEG): container finished" podID="37be5208-015f-4ab4-8bd0-f390d3729293" containerID="7f53e23ea8caaeff386481eb510505e2cdbff9e3b48f17e8ff5c776d81816509" exitCode=60 Oct 30 20:00:03.303149 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:03.303108 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" event={"ID":"37be5208-015f-4ab4-8bd0-f390d3729293","Type":"ContainerDied","Data":"7f53e23ea8caaeff386481eb510505e2cdbff9e3b48f17e8ff5c776d81816509"} Oct 30 20:00:03.303343 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:03.303329 4433 scope.go:122] "RemoveContainer" containerID="7f53e23ea8caaeff386481eb510505e2cdbff9e3b48f17e8ff5c776d81816509" Oct 30 20:00:03.303541 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:03.303521 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-ffddn_odh-ai-gateway-infra(37be5208-015f-4ab4-8bd0-f390d3729293)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" podUID="37be5208-015f-4ab4-8bd0-f390d3729293" Oct 30 20:00:03.304819 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:03.304794 4433 generic.go:356] "Generic (PLEG): container finished" podID="ded326eb-7db6-432f-8427-68833723a5fa" containerID="16c34239608d8cb3377b3a1c51e4227b4909ab0053b27ee012856d7190c717fc" exitCode=60 Oct 30 20:00:03.304937 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:03.304866 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" event={"ID":"ded326eb-7db6-432f-8427-68833723a5fa","Type":"ContainerDied","Data":"16c34239608d8cb3377b3a1c51e4227b4909ab0053b27ee012856d7190c717fc"} Oct 30 20:00:03.305113 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:03.305096 4433 scope.go:122] "RemoveContainer" containerID="16c34239608d8cb3377b3a1c51e4227b4909ab0053b27ee012856d7190c717fc" Oct 30 20:00:03.305326 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:03.305308 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6_odh-ai-gateway-infra(ded326eb-7db6-432f-8427-68833723a5fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" podUID="ded326eb-7db6-432f-8427-68833723a5fa" Oct 30 20:00:03.306703 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:03.306678 4433 generic.go:356] "Generic (PLEG): container finished" podID="2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" containerID="b4a443331fa7479a02d4169f87370ab79998cb5f155d93f0f2ac77660920cf9a" exitCode=60 Oct 30 20:00:03.306791 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:03.306731 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" event={"ID":"2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac","Type":"ContainerDied","Data":"b4a443331fa7479a02d4169f87370ab79998cb5f155d93f0f2ac77660920cf9a"} Oct 30 20:00:03.307108 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:03.307091 4433 scope.go:122] "RemoveContainer" containerID="b4a443331fa7479a02d4169f87370ab79998cb5f155d93f0f2ac77660920cf9a" Oct 30 20:00:03.307349 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:03.307328 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d_odh-ai-gateway-infra(2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" podUID="2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" Oct 30 20:00:03.330738 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:03.330720 4433 scope.go:122] "RemoveContainer" containerID="b1ee6e7e9672968627bb4e4bc6367e38e44ab8d104a7f5ec37076d0a1baa1b13" Oct 30 20:00:03.343046 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:03.343030 4433 scope.go:122] "RemoveContainer" containerID="fc00e6c3dc35b3bb27d31fec87ec12fbb34df0c63004d678ac9d006758823905" Oct 30 20:00:03.355612 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:03.355595 4433 scope.go:122] "RemoveContainer" containerID="e51812bc819389148641e33a08f5a7bc4be765d8e207a24dcc9851002d8a5f3b" Oct 30 20:00:04.214618 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.214588 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:04.337419 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.332851 4433 scope.go:122] "RemoveContainer" containerID="f804b6c9c8afe3df8f1908f76ff8a542f6f58f5a48494290e678aa3fda7d5282" Oct 30 20:00:04.337419 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:04.333252 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn_odh-ai-gateway-infra(56448e26-174f-4edd-96a7-87cbd714ad8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" podUID="56448e26-174f-4edd-96a7-87cbd714ad8e" Oct 30 20:00:04.337419 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.336740 4433 scope.go:122] "RemoveContainer" containerID="7f53e23ea8caaeff386481eb510505e2cdbff9e3b48f17e8ff5c776d81816509" Oct 30 20:00:04.337419 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:04.337050 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-ffddn_odh-ai-gateway-infra(37be5208-015f-4ab4-8bd0-f390d3729293)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" podUID="37be5208-015f-4ab4-8bd0-f390d3729293" Oct 30 20:00:04.339696 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.339675 4433 scope.go:122] "RemoveContainer" containerID="16c34239608d8cb3377b3a1c51e4227b4909ab0053b27ee012856d7190c717fc" Oct 30 20:00:04.339914 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:04.339895 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6_odh-ai-gateway-infra(ded326eb-7db6-432f-8427-68833723a5fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" podUID="ded326eb-7db6-432f-8427-68833723a5fa" Oct 30 20:00:04.341926 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.341899 4433 generic.go:356] "Generic (PLEG): container finished" podID="f1e0904a-f62b-4828-b130-fdef755b3831" containerID="b7565739b966ce2722d086cd53855837ae23c0b7ee2258b0125eb27961369afb" exitCode=0 Oct 30 20:00:04.342040 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.341957 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" event={"ID":"f1e0904a-f62b-4828-b130-fdef755b3831","Type":"ContainerDied","Data":"b7565739b966ce2722d086cd53855837ae23c0b7ee2258b0125eb27961369afb"} Oct 30 20:00:04.342040 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.342012 4433 scope.go:122] "RemoveContainer" containerID="035ca861208051e75bb966b5b5cf706a57efee09bd6075ffa690bf847e6f8330" Oct 30 20:00:04.346133 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.345537 4433 scope.go:122] "RemoveContainer" containerID="b4a443331fa7479a02d4169f87370ab79998cb5f155d93f0f2ac77660920cf9a" Oct 30 20:00:04.346133 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:04.345739 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d_odh-ai-gateway-infra(2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" podUID="2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" Oct 30 20:00:04.400607 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.400577 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 20:00:04.493938 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.493913 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-credential-socket\") pod \"f1e0904a-f62b-4828-b130-fdef755b3831\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " Oct 30 20:00:04.494081 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.493946 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-workload-certs\") pod \"f1e0904a-f62b-4828-b130-fdef755b3831\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " Oct 30 20:00:04.494081 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.493972 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-istio-data\") pod \"f1e0904a-f62b-4828-b130-fdef755b3831\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " Oct 30 20:00:04.494081 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.494062 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1e0904a-f62b-4828-b130-fdef755b3831-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e0904a-f62b-4828-b130-fdef755b3831-istio-podinfo\") pod \"f1e0904a-f62b-4828-b130-fdef755b3831\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " Oct 30 20:00:04.494081 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.494080 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-istio-envoy\") pod \"f1e0904a-f62b-4828-b130-fdef755b3831\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " Oct 30 20:00:04.494281 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.494123 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e0904a-f62b-4828-b130-fdef755b3831-kube-api-access-vxxcq\" (UniqueName: \"kubernetes.io/projected/f1e0904a-f62b-4828-b130-fdef755b3831-kube-api-access-vxxcq\") pod \"f1e0904a-f62b-4828-b130-fdef755b3831\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " Oct 30 20:00:04.494281 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.494156 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1e0904a-f62b-4828-b130-fdef755b3831-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e0904a-f62b-4828-b130-fdef755b3831-istiod-ca-cert\") pod \"f1e0904a-f62b-4828-b130-fdef755b3831\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " Oct 30 20:00:04.494281 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.494195 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-workload-socket\") pod \"f1e0904a-f62b-4828-b130-fdef755b3831\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " Oct 30 20:00:04.494281 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.494219 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-credential-socket" pod "f1e0904a-f62b-4828-b130-fdef755b3831" (UID: "f1e0904a-f62b-4828-b130-fdef755b3831"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:04.494281 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.494232 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-workload-certs" pod "f1e0904a-f62b-4828-b130-fdef755b3831" (UID: "f1e0904a-f62b-4828-b130-fdef755b3831"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:04.494527 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.494291 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e0904a-f62b-4828-b130-fdef755b3831-istio-token\" (UniqueName: \"kubernetes.io/projected/f1e0904a-f62b-4828-b130-fdef755b3831-istio-token\") pod \"f1e0904a-f62b-4828-b130-fdef755b3831\" (UID: \"f1e0904a-f62b-4828-b130-fdef755b3831\") " Oct 30 20:00:04.494527 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.494479 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-istio-data" pod "f1e0904a-f62b-4828-b130-fdef755b3831" (UID: "f1e0904a-f62b-4828-b130-fdef755b3831"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:04.494634 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.494535 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-workload-socket" pod "f1e0904a-f62b-4828-b130-fdef755b3831" (UID: "f1e0904a-f62b-4828-b130-fdef755b3831"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:04.494634 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.494574 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e0904a-f62b-4828-b130-fdef755b3831-istiod-ca-cert" pod "f1e0904a-f62b-4828-b130-fdef755b3831" (UID: "f1e0904a-f62b-4828-b130-fdef755b3831"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:04.494763 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.494746 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:04.494810 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.494772 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:04.494810 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.494789 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:04.494810 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.494805 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e0904a-f62b-4828-b130-fdef755b3831-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:04.494933 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.494819 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:04.496226 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.496196 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1e0904a-f62b-4828-b130-fdef755b3831-istio-podinfo" pod "f1e0904a-f62b-4828-b130-fdef755b3831" (UID: "f1e0904a-f62b-4828-b130-fdef755b3831"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:00:04.496494 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.496473 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e0904a-f62b-4828-b130-fdef755b3831-istio-token" pod "f1e0904a-f62b-4828-b130-fdef755b3831" (UID: "f1e0904a-f62b-4828-b130-fdef755b3831"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:04.496659 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.496634 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-istio-envoy" pod "f1e0904a-f62b-4828-b130-fdef755b3831" (UID: "f1e0904a-f62b-4828-b130-fdef755b3831"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:04.496757 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.496731 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e0904a-f62b-4828-b130-fdef755b3831-kube-api-access-vxxcq" pod "f1e0904a-f62b-4828-b130-fdef755b3831" (UID: "f1e0904a-f62b-4828-b130-fdef755b3831"). InnerVolumeSpecName "kube-api-access-vxxcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:04.517638 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.517616 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:04.596211 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.596185 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e0904a-f62b-4828-b130-fdef755b3831-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:04.596211 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.596208 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e0904a-f62b-4828-b130-fdef755b3831-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:04.596351 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.596218 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxxcq\" (UniqueName: \"kubernetes.io/projected/f1e0904a-f62b-4828-b130-fdef755b3831-kube-api-access-vxxcq\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:04.596351 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.596227 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1e0904a-f62b-4828-b130-fdef755b3831-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:04.661221 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.661159 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb"] Oct 30 20:00:04.672044 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.672018 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz"] Oct 30 20:00:04.763833 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:04.763745 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:05.065402 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:05.065363 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:05.353470 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:05.353390 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" Oct 30 20:00:05.353470 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:05.353415 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd" event={"ID":"f1e0904a-f62b-4828-b130-fdef755b3831","Type":"ContainerDied","Data":"8bc8601576984873e5b22667d78f94e30b4487b02d94c56708d0efc1e5ab722a"} Oct 30 20:00:05.353905 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:05.353477 4433 scope.go:122] "RemoveContainer" containerID="b7565739b966ce2722d086cd53855837ae23c0b7ee2258b0125eb27961369afb" Oct 30 20:00:05.397625 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:05.397594 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:05.521796 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:05.521756 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd"] Oct 30 20:00:05.525510 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:05.525480 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b0c827-openshift-default-67c96f9ddf-x62nd"] Oct 30 20:00:06.535658 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:06.535608 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:06.781263 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:06.781150 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj"] Oct 30 20:00:06.781502 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:06.781360 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" podUID="d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" containerName="istio-proxy" containerID="cri-o://4048e5777e14170ad9a2a0cbe7ce266ec9320a6aa26aac737f874db1ec5ee462" gracePeriod=30 Oct 30 20:00:06.789284 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:06.789222 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:07.109420 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:07.109389 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:07.213873 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:07.213842 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e0904a-f62b-4828-b130-fdef755b3831" path="/var/lib/kubelet/pods/f1e0904a-f62b-4828-b130-fdef755b3831/volumes" Oct 30 20:00:09.955116 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:09.955072 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:10.463605 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:10.463573 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:10.923619 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:10.923590 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:11.579735 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:11.579703 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd"] Oct 30 20:00:11.693784 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:11.693748 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh"] Oct 30 20:00:11.849328 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:11.849261 4433 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:00:11.849328 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:11.849312 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" podUID="d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 20:00:12.083420 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.083380 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:12.135927 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.135901 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 20:00:12.225233 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.225203 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-workload-socket\") pod \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " Oct 30 20:00:12.225429 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.225244 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-data\") pod \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " Oct 30 20:00:12.225429 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.225303 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-workload-certs\") pod \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " Oct 30 20:00:12.225429 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.225372 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-credential-socket\") pod \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " Oct 30 20:00:12.225429 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.225398 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-token\" (UniqueName: \"kubernetes.io/projected/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-token\") pod \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " Oct 30 20:00:12.225429 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.225420 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istiod-ca-cert\") pod \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " Oct 30 20:00:12.225702 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.225489 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-kube-api-access-5swxx\" (UniqueName: \"kubernetes.io/projected/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-kube-api-access-5swxx\") pod \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " Oct 30 20:00:12.225702 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.225503 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-workload-socket" pod "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" (UID: "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:12.225702 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.225523 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-data" pod "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" (UID: "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:12.225702 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.225542 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-podinfo\") pod \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " Oct 30 20:00:12.225702 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.225575 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-envoy\") pod \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\" (UID: \"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4\") " Oct 30 20:00:12.225702 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.225635 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-credential-socket" pod "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" (UID: "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:12.225702 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.225668 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-workload-certs" pod "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" (UID: "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:12.226057 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.225920 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istiod-ca-cert" pod "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" (UID: "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:12.226057 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.226026 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:12.226057 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.226046 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:12.226178 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.226060 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:12.226178 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.226074 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:12.226178 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.226087 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:12.227739 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.227710 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-kube-api-access-5swxx" pod "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" (UID: "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4"). InnerVolumeSpecName "kube-api-access-5swxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:12.227872 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.227748 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-podinfo" pod "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" (UID: "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:00:12.228026 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.227995 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-envoy" pod "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" (UID: "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:12.228254 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.228234 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-token" pod "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" (UID: "d00c90c8-e436-49e8-aee6-e5f4c7a38cb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:12.321396 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.321361 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerName="main" containerID="cri-o://746f076b7eede5cf82e4a802bfdfc3e92a3befbab05753a6218103ce8c8d7a69" gracePeriod=45 Oct 30 20:00:12.327830 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.327805 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:12.327830 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.327832 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5swxx\" (UniqueName: \"kubernetes.io/projected/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-kube-api-access-5swxx\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:12.328020 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.327842 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:12.328020 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.327854 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:12.333427 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.333403 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:12.397000 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.396924 4433 generic.go:356] "Generic (PLEG): container finished" podID="d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" containerID="4048e5777e14170ad9a2a0cbe7ce266ec9320a6aa26aac737f874db1ec5ee462" exitCode=0 Oct 30 20:00:12.397118 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.396994 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" Oct 30 20:00:12.397118 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.397006 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" event={"ID":"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4","Type":"ContainerDied","Data":"4048e5777e14170ad9a2a0cbe7ce266ec9320a6aa26aac737f874db1ec5ee462"} Oct 30 20:00:12.397118 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.397057 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj" event={"ID":"d00c90c8-e436-49e8-aee6-e5f4c7a38cb4","Type":"ContainerDied","Data":"38914cf7042f93a020c7c2b851a042ebdd6a107ca94647c8440c851718e7dc71"} Oct 30 20:00:12.397118 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.397078 4433 scope.go:122] "RemoveContainer" containerID="4048e5777e14170ad9a2a0cbe7ce266ec9320a6aa26aac737f874db1ec5ee462" Oct 30 20:00:12.434538 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.434521 4433 scope.go:122] "RemoveContainer" containerID="4048e5777e14170ad9a2a0cbe7ce266ec9320a6aa26aac737f874db1ec5ee462" Oct 30 20:00:12.434850 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:12.434825 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4048e5777e14170ad9a2a0cbe7ce266ec9320a6aa26aac737f874db1ec5ee462\": container with ID starting with 4048e5777e14170ad9a2a0cbe7ce266ec9320a6aa26aac737f874db1ec5ee462 not found: ID does not exist" containerID="4048e5777e14170ad9a2a0cbe7ce266ec9320a6aa26aac737f874db1ec5ee462" Oct 30 20:00:12.434916 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.434861 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4048e5777e14170ad9a2a0cbe7ce266ec9320a6aa26aac737f874db1ec5ee462"} err="failed to get container status \"4048e5777e14170ad9a2a0cbe7ce266ec9320a6aa26aac737f874db1ec5ee462\": rpc error: code = NotFound desc = could not find container \"4048e5777e14170ad9a2a0cbe7ce266ec9320a6aa26aac737f874db1ec5ee462\": container with ID starting with 4048e5777e14170ad9a2a0cbe7ce266ec9320a6aa26aac737f874db1ec5ee462 not found: ID does not exist" Oct 30 20:00:12.438699 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.438673 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": dial tcp 10.134.0.72:8000: connect: connection refused" Oct 30 20:00:12.567337 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.567306 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj"] Oct 30 20:00:12.571337 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.571313 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-z87tj"] Oct 30 20:00:12.572472 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.572450 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 20:00:12.733095 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.732675 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-tmp-dir\") pod \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " Oct 30 20:00:12.733095 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.732796 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-home\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-home\") pod \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " Oct 30 20:00:12.733095 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.732844 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c9c64c5-157e-4750-9a42-3c2b7f15e741-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c9c64c5-157e-4750-9a42-3c2b7f15e741-tls-certs\") pod \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " Oct 30 20:00:12.733095 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.732869 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-dshm\") pod \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " Oct 30 20:00:12.733095 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.732914 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9c64c5-157e-4750-9a42-3c2b7f15e741-kube-api-access-65fpl\" (UniqueName: \"kubernetes.io/projected/7c9c64c5-157e-4750-9a42-3c2b7f15e741-kube-api-access-65fpl\") pod \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " Oct 30 20:00:12.733095 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.732957 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-model-cache\") pod \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\" (UID: \"7c9c64c5-157e-4750-9a42-3c2b7f15e741\") " Oct 30 20:00:12.733095 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.733020 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-tmp-dir" pod "7c9c64c5-157e-4750-9a42-3c2b7f15e741" (UID: "7c9c64c5-157e-4750-9a42-3c2b7f15e741"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:12.733859 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.733317 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-model-cache" pod "7c9c64c5-157e-4750-9a42-3c2b7f15e741" (UID: "7c9c64c5-157e-4750-9a42-3c2b7f15e741"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:12.733859 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.733546 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-home" pod "7c9c64c5-157e-4750-9a42-3c2b7f15e741" (UID: "7c9c64c5-157e-4750-9a42-3c2b7f15e741"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:12.733859 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.733831 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-tmp-dir\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:12.733859 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.733852 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-home\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:12.734106 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.733865 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-model-cache\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:12.736145 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.736114 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9c64c5-157e-4750-9a42-3c2b7f15e741-kube-api-access-65fpl" pod "7c9c64c5-157e-4750-9a42-3c2b7f15e741" (UID: "7c9c64c5-157e-4750-9a42-3c2b7f15e741"). InnerVolumeSpecName "kube-api-access-65fpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:12.736429 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.736405 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c9c64c5-157e-4750-9a42-3c2b7f15e741-tls-certs" pod "7c9c64c5-157e-4750-9a42-3c2b7f15e741" (UID: "7c9c64c5-157e-4750-9a42-3c2b7f15e741"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:12.736523 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.736469 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-dshm" pod "7c9c64c5-157e-4750-9a42-3c2b7f15e741" (UID: "7c9c64c5-157e-4750-9a42-3c2b7f15e741"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:12.834694 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.834669 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c9c64c5-157e-4750-9a42-3c2b7f15e741-tls-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:12.834694 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.834694 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c9c64c5-157e-4750-9a42-3c2b7f15e741-dshm\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:12.834851 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:12.834704 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65fpl\" (UniqueName: \"kubernetes.io/projected/7c9c64c5-157e-4750-9a42-3c2b7f15e741-kube-api-access-65fpl\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:13.221115 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:13.221079 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" path="/var/lib/kubelet/pods/d00c90c8-e436-49e8-aee6-e5f4c7a38cb4/volumes" Oct 30 20:00:13.405776 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:13.405736 4433 generic.go:356] "Generic (PLEG): container finished" podID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerID="746f076b7eede5cf82e4a802bfdfc3e92a3befbab05753a6218103ce8c8d7a69" exitCode=0 Oct 30 20:00:13.405942 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:13.405786 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" Oct 30 20:00:13.405942 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:13.405822 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" event={"ID":"7c9c64c5-157e-4750-9a42-3c2b7f15e741","Type":"ContainerDied","Data":"746f076b7eede5cf82e4a802bfdfc3e92a3befbab05753a6218103ce8c8d7a69"} Oct 30 20:00:13.405942 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:13.405868 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j" event={"ID":"7c9c64c5-157e-4750-9a42-3c2b7f15e741","Type":"ContainerDied","Data":"5ddf4385aa5381570f583a74fc486ce28ef3e7704f1389d1881e3fd175451c4d"} Oct 30 20:00:13.405942 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:13.405886 4433 scope.go:122] "RemoveContainer" containerID="746f076b7eede5cf82e4a802bfdfc3e92a3befbab05753a6218103ce8c8d7a69" Oct 30 20:00:13.433855 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:13.433154 4433 scope.go:122] "RemoveContainer" containerID="746f076b7eede5cf82e4a802bfdfc3e92a3befbab05753a6218103ce8c8d7a69" Oct 30 20:00:13.451346 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:13.450307 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746f076b7eede5cf82e4a802bfdfc3e92a3befbab05753a6218103ce8c8d7a69\": container with ID starting with 746f076b7eede5cf82e4a802bfdfc3e92a3befbab05753a6218103ce8c8d7a69 not found: ID does not exist" containerID="746f076b7eede5cf82e4a802bfdfc3e92a3befbab05753a6218103ce8c8d7a69" Oct 30 20:00:13.451346 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:13.450369 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746f076b7eede5cf82e4a802bfdfc3e92a3befbab05753a6218103ce8c8d7a69"} err="failed to get container status \"746f076b7eede5cf82e4a802bfdfc3e92a3befbab05753a6218103ce8c8d7a69\": rpc error: code = NotFound desc = could not find container \"746f076b7eede5cf82e4a802bfdfc3e92a3befbab05753a6218103ce8c8d7a69\": container with ID starting with 746f076b7eede5cf82e4a802bfdfc3e92a3befbab05753a6218103ce8c8d7a69 not found: ID does not exist" Oct 30 20:00:13.571535 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:13.571496 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j"] Oct 30 20:00:13.574971 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:13.574944 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/distinct-w6-5f55cd-kserve-599869cd77-l878j"] Oct 30 20:00:14.206826 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:14.206801 4433 scope.go:122] "RemoveContainer" containerID="7f53e23ea8caaeff386481eb510505e2cdbff9e3b48f17e8ff5c776d81816509" Oct 30 20:00:14.207223 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:14.207011 4433 scope.go:122] "RemoveContainer" containerID="b4a443331fa7479a02d4169f87370ab79998cb5f155d93f0f2ac77660920cf9a" Oct 30 20:00:14.759729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:14.759626 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:15.006059 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:15.006020 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:15.206513 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:15.206489 4433 scope.go:122] "RemoveContainer" containerID="f804b6c9c8afe3df8f1908f76ff8a542f6f58f5a48494290e678aa3fda7d5282" Oct 30 20:00:15.214374 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:15.214352 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" path="/var/lib/kubelet/pods/7c9c64c5-157e-4750-9a42-3c2b7f15e741/volumes" Oct 30 20:00:15.429665 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:15.429634 4433 generic.go:356] "Generic (PLEG): container finished" podID="2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" containerID="ac94933a9ce41ee3f80bf00e541e582efa249c825f95ae980be1a18732e5efcf" exitCode=60 Oct 30 20:00:15.429786 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:15.429708 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" event={"ID":"2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac","Type":"ContainerDied","Data":"ac94933a9ce41ee3f80bf00e541e582efa249c825f95ae980be1a18732e5efcf"} Oct 30 20:00:15.429841 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:15.429797 4433 scope.go:122] "RemoveContainer" containerID="b4a443331fa7479a02d4169f87370ab79998cb5f155d93f0f2ac77660920cf9a" Oct 30 20:00:15.430182 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:15.430163 4433 scope.go:122] "RemoveContainer" containerID="ac94933a9ce41ee3f80bf00e541e582efa249c825f95ae980be1a18732e5efcf" Oct 30 20:00:15.430407 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:15.430380 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d_odh-ai-gateway-infra(2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" podUID="2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" Oct 30 20:00:15.431702 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:15.431681 4433 generic.go:356] "Generic (PLEG): container finished" podID="37be5208-015f-4ab4-8bd0-f390d3729293" containerID="a3b63425c4b2b58b54d7ab29f8517317e551b0fcf8218616a77ffd9c144635d7" exitCode=60 Oct 30 20:00:15.431792 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:15.431751 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" event={"ID":"37be5208-015f-4ab4-8bd0-f390d3729293","Type":"ContainerDied","Data":"a3b63425c4b2b58b54d7ab29f8517317e551b0fcf8218616a77ffd9c144635d7"} Oct 30 20:00:15.432059 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:15.432045 4433 scope.go:122] "RemoveContainer" containerID="a3b63425c4b2b58b54d7ab29f8517317e551b0fcf8218616a77ffd9c144635d7" Oct 30 20:00:15.432246 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:15.432230 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-ffddn_odh-ai-gateway-infra(37be5208-015f-4ab4-8bd0-f390d3729293)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" podUID="37be5208-015f-4ab4-8bd0-f390d3729293" Oct 30 20:00:15.442411 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:15.442393 4433 scope.go:122] "RemoveContainer" containerID="7f53e23ea8caaeff386481eb510505e2cdbff9e3b48f17e8ff5c776d81816509" Oct 30 20:00:16.207142 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.207116 4433 scope.go:122] "RemoveContainer" containerID="16c34239608d8cb3377b3a1c51e4227b4909ab0053b27ee012856d7190c717fc" Oct 30 20:00:16.441157 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.441128 4433 generic.go:356] "Generic (PLEG): container finished" podID="56448e26-174f-4edd-96a7-87cbd714ad8e" containerID="954bc5cea79132928ea1ac9dfc7c8c7c2f0e9242761e11008892b9ccbe275e0c" exitCode=60 Oct 30 20:00:16.441556 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.441202 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" event={"ID":"56448e26-174f-4edd-96a7-87cbd714ad8e","Type":"ContainerDied","Data":"954bc5cea79132928ea1ac9dfc7c8c7c2f0e9242761e11008892b9ccbe275e0c"} Oct 30 20:00:16.441556 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.441258 4433 scope.go:122] "RemoveContainer" containerID="f804b6c9c8afe3df8f1908f76ff8a542f6f58f5a48494290e678aa3fda7d5282" Oct 30 20:00:16.441655 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.441620 4433 scope.go:122] "RemoveContainer" containerID="954bc5cea79132928ea1ac9dfc7c8c7c2f0e9242761e11008892b9ccbe275e0c" Oct 30 20:00:16.441937 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:16.441914 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn_odh-ai-gateway-infra(56448e26-174f-4edd-96a7-87cbd714ad8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" podUID="56448e26-174f-4edd-96a7-87cbd714ad8e" Oct 30 20:00:16.564389 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.564359 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d"] Oct 30 20:00:16.580729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.580701 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" podUID="34f8f88e-3537-4e62-ae87-b12ec3812f62" containerName="payload-processing" containerID="cri-o://b03cdc10b9b3c337f6ee33f18f184de3843bff71f25c515def64405888e84828" gracePeriod=25 Oct 30 20:00:16.682713 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.682644 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn"] Oct 30 20:00:16.694777 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.694727 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" podUID="e60baa7a-9536-4dd5-b59c-d042cd41b798" containerName="payload-pre-processing" containerID="cri-o://7aa07c6e89eafcb90edfce328f8c892e08d8552e831a34b78f005fb76b0af478" gracePeriod=25 Oct 30 20:00:16.723165 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.723118 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" Oct 30 20:00:16.797097 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.797063 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:16.844353 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.844330 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" Oct 30 20:00:16.897967 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.897931 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac-maas-api-service-ca\") pod \"2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac\" (UID: \"2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac\") " Oct 30 20:00:16.898192 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.898167 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac-kube-api-access-q5nsb\" (UniqueName: \"kubernetes.io/projected/2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac-kube-api-access-q5nsb\") pod \"2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac\" (UID: \"2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac\") " Oct 30 20:00:16.898332 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.898304 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac-maas-api-service-ca" pod "2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" (UID: "2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:16.898832 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.898812 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac-maas-api-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:16.900205 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.900187 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac-kube-api-access-q5nsb" pod "2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" (UID: "2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac"). InnerVolumeSpecName "kube-api-access-q5nsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:16.999942 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:16.999914 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37be5208-015f-4ab4-8bd0-f390d3729293-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37be5208-015f-4ab4-8bd0-f390d3729293-maas-api-service-ca\") pod \"37be5208-015f-4ab4-8bd0-f390d3729293\" (UID: \"37be5208-015f-4ab4-8bd0-f390d3729293\") " Oct 30 20:00:17.000195 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.000176 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37be5208-015f-4ab4-8bd0-f390d3729293-kube-api-access-sbvhc\" (UniqueName: \"kubernetes.io/projected/37be5208-015f-4ab4-8bd0-f390d3729293-kube-api-access-sbvhc\") pod \"37be5208-015f-4ab4-8bd0-f390d3729293\" (UID: \"37be5208-015f-4ab4-8bd0-f390d3729293\") " Oct 30 20:00:17.000279 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.000198 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37be5208-015f-4ab4-8bd0-f390d3729293-maas-api-service-ca" pod "37be5208-015f-4ab4-8bd0-f390d3729293" (UID: "37be5208-015f-4ab4-8bd0-f390d3729293"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:17.000620 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.000603 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37be5208-015f-4ab4-8bd0-f390d3729293-maas-api-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.000682 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.000626 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5nsb\" (UniqueName: \"kubernetes.io/projected/2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac-kube-api-access-q5nsb\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.002095 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.002071 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37be5208-015f-4ab4-8bd0-f390d3729293-kube-api-access-sbvhc" pod "37be5208-015f-4ab4-8bd0-f390d3729293" (UID: "37be5208-015f-4ab4-8bd0-f390d3729293"). InnerVolumeSpecName "kube-api-access-sbvhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:17.071703 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.071677 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:17.102160 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.102137 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbvhc\" (UniqueName: \"kubernetes.io/projected/37be5208-015f-4ab4-8bd0-f390d3729293-kube-api-access-sbvhc\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.254525 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:17.254466 4433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37be5208_015f_4ab4_8bd0_f390d3729293.slice\": RecentStats: unable to find data in memory cache]" Oct 30 20:00:17.450931 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.450900 4433 generic.go:356] "Generic (PLEG): container finished" podID="ded326eb-7db6-432f-8427-68833723a5fa" containerID="72dec5f1c6ca761e0db92ae2d238a8f104392bcb17027f786f2e9ab31a6d567a" exitCode=60 Oct 30 20:00:17.451374 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.450974 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" event={"ID":"ded326eb-7db6-432f-8427-68833723a5fa","Type":"ContainerDied","Data":"72dec5f1c6ca761e0db92ae2d238a8f104392bcb17027f786f2e9ab31a6d567a"} Oct 30 20:00:17.451374 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.451060 4433 scope.go:122] "RemoveContainer" containerID="16c34239608d8cb3377b3a1c51e4227b4909ab0053b27ee012856d7190c717fc" Oct 30 20:00:17.451374 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.451366 4433 scope.go:122] "RemoveContainer" containerID="72dec5f1c6ca761e0db92ae2d238a8f104392bcb17027f786f2e9ab31a6d567a" Oct 30 20:00:17.451679 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:17.451659 4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6_odh-ai-gateway-infra(ded326eb-7db6-432f-8427-68833723a5fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" podUID="ded326eb-7db6-432f-8427-68833723a5fa" Oct 30 20:00:17.452369 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.452353 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" Oct 30 20:00:17.452464 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.452404 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d" event={"ID":"2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac","Type":"ContainerDied","Data":"afec48fb6dd4246f211e7504af64771dfa2696da5558d7385f75687edd4b1aca"} Oct 30 20:00:17.455391 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.455370 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" event={"ID":"37be5208-015f-4ab4-8bd0-f390d3729293","Type":"ContainerDied","Data":"d1ccbc182c7e3f799598346dc1429b3c871491127e2ba6e9588ef8f066571886"} Oct 30 20:00:17.455480 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.455377 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn" Oct 30 20:00:17.463455 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.463427 4433 scope.go:122] "RemoveContainer" containerID="ac94933a9ce41ee3f80bf00e541e582efa249c825f95ae980be1a18732e5efcf" Oct 30 20:00:17.473035 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.473019 4433 scope.go:122] "RemoveContainer" containerID="a3b63425c4b2b58b54d7ab29f8517317e551b0fcf8218616a77ffd9c144635d7" Oct 30 20:00:17.584483 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.584459 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn"] Oct 30 20:00:17.682765 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.682717 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d"] Oct 30 20:00:17.688025 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.687974 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a3f9b7-29889840-pvv9d"] Oct 30 20:00:17.743268 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.743246 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" Oct 30 20:00:17.802274 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.802249 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn"] Oct 30 20:00:17.807957 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.807935 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-ffddn"] Oct 30 20:00:17.909411 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.909349 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56448e26-174f-4edd-96a7-87cbd714ad8e-kube-api-access-kg77r\" (UniqueName: \"kubernetes.io/projected/56448e26-174f-4edd-96a7-87cbd714ad8e-kube-api-access-kg77r\") pod \"56448e26-174f-4edd-96a7-87cbd714ad8e\" (UID: \"56448e26-174f-4edd-96a7-87cbd714ad8e\") " Oct 30 20:00:17.909530 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.909421 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56448e26-174f-4edd-96a7-87cbd714ad8e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56448e26-174f-4edd-96a7-87cbd714ad8e-maas-api-service-ca\") pod \"56448e26-174f-4edd-96a7-87cbd714ad8e\" (UID: \"56448e26-174f-4edd-96a7-87cbd714ad8e\") " Oct 30 20:00:17.909798 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.909777 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56448e26-174f-4edd-96a7-87cbd714ad8e-maas-api-service-ca" pod "56448e26-174f-4edd-96a7-87cbd714ad8e" (UID: "56448e26-174f-4edd-96a7-87cbd714ad8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:17.909908 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.909893 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56448e26-174f-4edd-96a7-87cbd714ad8e-maas-api-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.911292 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:17.911273 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56448e26-174f-4edd-96a7-87cbd714ad8e-kube-api-access-kg77r" pod "56448e26-174f-4edd-96a7-87cbd714ad8e" (UID: "56448e26-174f-4edd-96a7-87cbd714ad8e"). InnerVolumeSpecName "kube-api-access-kg77r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:18.011140 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:18.011110 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg77r\" (UniqueName: \"kubernetes.io/projected/56448e26-174f-4edd-96a7-87cbd714ad8e-kube-api-access-kg77r\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.464804 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:18.464776 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" Oct 30 20:00:18.464804 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:18.464793 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn" event={"ID":"56448e26-174f-4edd-96a7-87cbd714ad8e","Type":"ContainerDied","Data":"3685f727fd19ff04dc2761b1f53ff43c2ce024760371eeb19ebccd751407c672"} Oct 30 20:00:18.465340 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:18.464857 4433 scope.go:122] "RemoveContainer" containerID="954bc5cea79132928ea1ac9dfc7c8c7c2f0e9242761e11008892b9ccbe275e0c" Oct 30 20:00:18.587665 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:18.587638 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6"] Oct 30 20:00:18.638462 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:18.638432 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn"] Oct 30 20:00:18.644422 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:18.644396 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b5178f-29889840-cgpjn"] Oct 30 20:00:18.731258 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:18.731204 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" Oct 30 20:00:18.817803 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:18.817776 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded326eb-7db6-432f-8427-68833723a5fa-kube-api-access-98bl9\" (UniqueName: \"kubernetes.io/projected/ded326eb-7db6-432f-8427-68833723a5fa-kube-api-access-98bl9\") pod \"ded326eb-7db6-432f-8427-68833723a5fa\" (UID: \"ded326eb-7db6-432f-8427-68833723a5fa\") " Oct 30 20:00:18.817946 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:18.817833 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded326eb-7db6-432f-8427-68833723a5fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded326eb-7db6-432f-8427-68833723a5fa-maas-api-service-ca\") pod \"ded326eb-7db6-432f-8427-68833723a5fa\" (UID: \"ded326eb-7db6-432f-8427-68833723a5fa\") " Oct 30 20:00:18.818210 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:18.818188 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded326eb-7db6-432f-8427-68833723a5fa-maas-api-service-ca" pod "ded326eb-7db6-432f-8427-68833723a5fa" (UID: "ded326eb-7db6-432f-8427-68833723a5fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:18.818351 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:18.818335 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded326eb-7db6-432f-8427-68833723a5fa-maas-api-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.819724 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:18.819703 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded326eb-7db6-432f-8427-68833723a5fa-kube-api-access-98bl9" pod "ded326eb-7db6-432f-8427-68833723a5fa" (UID: "ded326eb-7db6-432f-8427-68833723a5fa"). InnerVolumeSpecName "kube-api-access-98bl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:18.920196 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:18.920160 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98bl9\" (UniqueName: \"kubernetes.io/projected/ded326eb-7db6-432f-8427-68833723a5fa-kube-api-access-98bl9\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:19.214051 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:19.214022 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" path="/var/lib/kubelet/pods/2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac/volumes" Oct 30 20:00:19.214530 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:19.214507 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37be5208-015f-4ab4-8bd0-f390d3729293" path="/var/lib/kubelet/pods/37be5208-015f-4ab4-8bd0-f390d3729293/volumes" Oct 30 20:00:19.214953 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:19.214933 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56448e26-174f-4edd-96a7-87cbd714ad8e" path="/var/lib/kubelet/pods/56448e26-174f-4edd-96a7-87cbd714ad8e/volumes" Oct 30 20:00:19.474283 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:19.474215 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" Oct 30 20:00:19.474283 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:19.474212 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6" event={"ID":"ded326eb-7db6-432f-8427-68833723a5fa","Type":"ContainerDied","Data":"8f91a55ba0b62ba7ab0a181a7191d9713e94a7aee3b56ce83f543ca446157d42"} Oct 30 20:00:19.474770 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:19.474318 4433 scope.go:122] "RemoveContainer" containerID="72dec5f1c6ca761e0db92ae2d238a8f104392bcb17027f786f2e9ab31a6d567a" Oct 30 20:00:19.646280 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:19.646251 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6"] Oct 30 20:00:19.651583 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:19.651558 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f55cd-29889840-wggs6"] Oct 30 20:00:19.682381 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:19.682352 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" podUID="eafb6a50-b7db-47d6-b742-86b92bfef67d" containerName="main" containerID="cri-o://7c936addb1988b075839c20ebb39ef2a97566cec1425757ca505883408d161da" gracePeriod=45 Oct 30 20:00:19.696388 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:19.696361 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" containerID="cri-o://6984a105a08251dc735276133323bedf9256f5fe7f63b92f1cee9dd1f06c0098" gracePeriod=45 Oct 30 20:00:19.992322 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:19.992299 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 20:00:19.995448 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:19.995427 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 20:00:20.132963 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.132930 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eafb6a50-b7db-47d6-b742-86b92bfef67d-tls-certs\" (UniqueName: \"kubernetes.io/secret/eafb6a50-b7db-47d6-b742-86b92bfef67d-tls-certs\") pod \"eafb6a50-b7db-47d6-b742-86b92bfef67d\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " Oct 30 20:00:20.133170 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.133046 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-dshm\") pod \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " Oct 30 20:00:20.133170 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.133102 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-home\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-home\") pod \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " Oct 30 20:00:20.133290 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.133199 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafb6a50-b7db-47d6-b742-86b92bfef67d-kube-api-access-hr5qv\" (UniqueName: \"kubernetes.io/projected/eafb6a50-b7db-47d6-b742-86b92bfef67d-kube-api-access-hr5qv\") pod \"eafb6a50-b7db-47d6-b742-86b92bfef67d\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " Oct 30 20:00:20.133290 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.133260 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-kube-api-access-vps9b\" (UniqueName: \"kubernetes.io/projected/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-kube-api-access-vps9b\") pod \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " Oct 30 20:00:20.133402 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.133336 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-model-cache\") pod \"eafb6a50-b7db-47d6-b742-86b92bfef67d\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " Oct 30 20:00:20.133402 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.133361 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-home" pod "8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" (UID: "8c6dccf0-37fd-41f6-ae17-ab1e44c13c89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:20.133402 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.133377 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-model-cache\") pod \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.133444 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-tls-certs\") pod \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.133500 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-tmp-dir\") pod \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\" (UID: \"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89\") " Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.133537 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-dshm\") pod \"eafb6a50-b7db-47d6-b742-86b92bfef67d\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.133575 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-model-cache" pod "eafb6a50-b7db-47d6-b742-86b92bfef67d" (UID: "eafb6a50-b7db-47d6-b742-86b92bfef67d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.133593 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-home\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-home\") pod \"eafb6a50-b7db-47d6-b742-86b92bfef67d\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.133651 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-model-cache" pod "8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" (UID: "8c6dccf0-37fd-41f6-ae17-ab1e44c13c89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.133677 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-tmp-dir\") pod \"eafb6a50-b7db-47d6-b742-86b92bfef67d\" (UID: \"eafb6a50-b7db-47d6-b742-86b92bfef67d\") " Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.133769 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-home" pod "eafb6a50-b7db-47d6-b742-86b92bfef67d" (UID: "eafb6a50-b7db-47d6-b742-86b92bfef67d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.133844 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-tmp-dir" pod "8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" (UID: "8c6dccf0-37fd-41f6-ae17-ab1e44c13c89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.134076 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-tmp-dir" pod "eafb6a50-b7db-47d6-b742-86b92bfef67d" (UID: "eafb6a50-b7db-47d6-b742-86b92bfef67d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.134303 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-tmp-dir\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.134325 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-home\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.134337 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-tmp-dir\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.134348 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-home\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.134360 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-model-cache\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.134373 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-model-cache\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.136087 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafb6a50-b7db-47d6-b742-86b92bfef67d-kube-api-access-hr5qv" pod "eafb6a50-b7db-47d6-b742-86b92bfef67d" (UID: "eafb6a50-b7db-47d6-b742-86b92bfef67d"). InnerVolumeSpecName "kube-api-access-hr5qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.136339 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-dshm" pod "eafb6a50-b7db-47d6-b742-86b92bfef67d" (UID: "eafb6a50-b7db-47d6-b742-86b92bfef67d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.136380 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-tls-certs" pod "8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" (UID: "8c6dccf0-37fd-41f6-ae17-ab1e44c13c89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:20.137039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.136410 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eafb6a50-b7db-47d6-b742-86b92bfef67d-tls-certs" pod "eafb6a50-b7db-47d6-b742-86b92bfef67d" (UID: "eafb6a50-b7db-47d6-b742-86b92bfef67d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:20.138124 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.137509 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-dshm" pod "8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" (UID: "8c6dccf0-37fd-41f6-ae17-ab1e44c13c89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:20.138124 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.137802 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-kube-api-access-vps9b" pod "8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" (UID: "8c6dccf0-37fd-41f6-ae17-ab1e44c13c89"). InnerVolumeSpecName "kube-api-access-vps9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:20.155969 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.155947 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:20.249046 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.244025 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eafb6a50-b7db-47d6-b742-86b92bfef67d-tls-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.249046 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.244065 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-dshm\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.249046 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.244079 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr5qv\" (UniqueName: \"kubernetes.io/projected/eafb6a50-b7db-47d6-b742-86b92bfef67d-kube-api-access-hr5qv\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.249046 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.244093 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vps9b\" (UniqueName: \"kubernetes.io/projected/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-kube-api-access-vps9b\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.249046 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.244110 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89-tls-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.249046 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.244124 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eafb6a50-b7db-47d6-b742-86b92bfef67d-dshm\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.482784 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.482699 4433 generic.go:356] "Generic (PLEG): container finished" podID="eafb6a50-b7db-47d6-b742-86b92bfef67d" containerID="7c936addb1988b075839c20ebb39ef2a97566cec1425757ca505883408d161da" exitCode=0 Oct 30 20:00:20.482784 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.482745 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" Oct 30 20:00:20.483275 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.482783 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" event={"ID":"eafb6a50-b7db-47d6-b742-86b92bfef67d","Type":"ContainerDied","Data":"7c936addb1988b075839c20ebb39ef2a97566cec1425757ca505883408d161da"} Oct 30 20:00:20.483275 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.482828 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" event={"ID":"eafb6a50-b7db-47d6-b742-86b92bfef67d","Type":"ContainerDied","Data":"65f620b0f6620cbe3de7432530eebf14c38aab1afda582e00b21438eff458e0b"} Oct 30 20:00:20.483275 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.482850 4433 scope.go:122] "RemoveContainer" containerID="7c936addb1988b075839c20ebb39ef2a97566cec1425757ca505883408d161da" Oct 30 20:00:20.484468 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.484442 4433 generic.go:356] "Generic (PLEG): container finished" podID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerID="6984a105a08251dc735276133323bedf9256f5fe7f63b92f1cee9dd1f06c0098" exitCode=0 Oct 30 20:00:20.484589 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.484486 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" Oct 30 20:00:20.484667 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.484582 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" event={"ID":"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89","Type":"ContainerDied","Data":"6984a105a08251dc735276133323bedf9256f5fe7f63b92f1cee9dd1f06c0098"} Oct 30 20:00:20.484667 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.484629 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz" event={"ID":"8c6dccf0-37fd-41f6-ae17-ab1e44c13c89","Type":"ContainerDied","Data":"aaf9caedc551b2e438798d5a4330dffa50847c04feb1104b8f424c0901a682e3"} Oct 30 20:00:20.493872 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.493855 4433 scope.go:122] "RemoveContainer" containerID="7c936addb1988b075839c20ebb39ef2a97566cec1425757ca505883408d161da" Oct 30 20:00:20.494172 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:20.494147 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c936addb1988b075839c20ebb39ef2a97566cec1425757ca505883408d161da\": container with ID starting with 7c936addb1988b075839c20ebb39ef2a97566cec1425757ca505883408d161da not found: ID does not exist" containerID="7c936addb1988b075839c20ebb39ef2a97566cec1425757ca505883408d161da" Oct 30 20:00:20.494246 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.494180 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c936addb1988b075839c20ebb39ef2a97566cec1425757ca505883408d161da"} err="failed to get container status \"7c936addb1988b075839c20ebb39ef2a97566cec1425757ca505883408d161da\": rpc error: code = NotFound desc = could not find container \"7c936addb1988b075839c20ebb39ef2a97566cec1425757ca505883408d161da\": container with ID starting with 7c936addb1988b075839c20ebb39ef2a97566cec1425757ca505883408d161da not found: ID does not exist" Oct 30 20:00:20.494246 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.494195 4433 scope.go:122] "RemoveContainer" containerID="6984a105a08251dc735276133323bedf9256f5fe7f63b92f1cee9dd1f06c0098" Oct 30 20:00:20.502505 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.502489 4433 scope.go:122] "RemoveContainer" containerID="6984a105a08251dc735276133323bedf9256f5fe7f63b92f1cee9dd1f06c0098" Oct 30 20:00:20.502740 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:20.502717 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6984a105a08251dc735276133323bedf9256f5fe7f63b92f1cee9dd1f06c0098\": container with ID starting with 6984a105a08251dc735276133323bedf9256f5fe7f63b92f1cee9dd1f06c0098 not found: ID does not exist" containerID="6984a105a08251dc735276133323bedf9256f5fe7f63b92f1cee9dd1f06c0098" Oct 30 20:00:20.502833 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.502744 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6984a105a08251dc735276133323bedf9256f5fe7f63b92f1cee9dd1f06c0098"} err="failed to get container status \"6984a105a08251dc735276133323bedf9256f5fe7f63b92f1cee9dd1f06c0098\": rpc error: code = NotFound desc = could not find container \"6984a105a08251dc735276133323bedf9256f5fe7f63b92f1cee9dd1f06c0098\": container with ID starting with 6984a105a08251dc735276133323bedf9256f5fe7f63b92f1cee9dd1f06c0098 not found: ID does not exist" Oct 30 20:00:20.646514 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.646484 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz"] Oct 30 20:00:20.695706 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.695685 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-686689/simulator-w5-686689-kserve-8667946bf5-r2pmz"] Oct 30 20:00:20.761490 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.761429 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb"] Oct 30 20:00:20.765547 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.765528 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb"] Oct 30 20:00:20.926395 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:20.926361 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/premium-w5-686689-kserve-5984d96bb6-v2zxb" podUID="eafb6a50-b7db-47d6-b742-86b92bfef67d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 20:00:21.214489 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:21.214459 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" path="/var/lib/kubelet/pods/8c6dccf0-37fd-41f6-ae17-ab1e44c13c89/volumes" Oct 30 20:00:21.214930 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:21.214911 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded326eb-7db6-432f-8427-68833723a5fa" path="/var/lib/kubelet/pods/ded326eb-7db6-432f-8427-68833723a5fa/volumes" Oct 30 20:00:21.215372 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:21.215357 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eafb6a50-b7db-47d6-b742-86b92bfef67d" path="/var/lib/kubelet/pods/eafb6a50-b7db-47d6-b742-86b92bfef67d/volumes" Oct 30 20:00:21.352171 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:21.352136 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" podUID="34f8f88e-3537-4e62-ae87-b12ec3812f62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:00:21.352325 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:21.352209 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" podUID="e60baa7a-9536-4dd5-b59c-d042cd41b798" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:00:21.521151 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:21.521043 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:21.779238 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:21.779047 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:23.365236 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:23.365202 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:00:24.464544 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:24.464500 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:25.225073 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:25.225009 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:26.010499 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:26.010470 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:26.279418 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:26.279337 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:26.536037 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:26.535956 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:26.964063 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:26.963943 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:31.300146 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:31.300106 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:31.351969 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:31.351942 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" podUID="e60baa7a-9536-4dd5-b59c-d042cd41b798" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:00:31.352124 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:31.352002 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" podUID="34f8f88e-3537-4e62-ae87-b12ec3812f62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:00:33.622544 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:33.622505 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:33.880194 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:33.880111 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:34.365290 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:34.365242 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:36.080035 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:36.079969 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:36.359209 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:36.359133 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:36.847513 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:36.847489 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" Oct 30 20:00:36.949273 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:36.949210 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f8f88e-3537-4e62-ae87-b12ec3812f62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34f8f88e-3537-4e62-ae87-b12ec3812f62-plugins-config-volume\") pod \"34f8f88e-3537-4e62-ae87-b12ec3812f62\" (UID: \"34f8f88e-3537-4e62-ae87-b12ec3812f62\") " Oct 30 20:00:36.949389 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:36.949303 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f8f88e-3537-4e62-ae87-b12ec3812f62-kube-api-access-p6lkf\" (UniqueName: \"kubernetes.io/projected/34f8f88e-3537-4e62-ae87-b12ec3812f62-kube-api-access-p6lkf\") pod \"34f8f88e-3537-4e62-ae87-b12ec3812f62\" (UID: \"34f8f88e-3537-4e62-ae87-b12ec3812f62\") " Oct 30 20:00:36.949569 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:36.949550 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f8f88e-3537-4e62-ae87-b12ec3812f62-plugins-config-volume" pod "34f8f88e-3537-4e62-ae87-b12ec3812f62" (UID: "34f8f88e-3537-4e62-ae87-b12ec3812f62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:36.949752 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:36.949730 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34f8f88e-3537-4e62-ae87-b12ec3812f62-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:36.951373 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:36.951354 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f8f88e-3537-4e62-ae87-b12ec3812f62-kube-api-access-p6lkf" pod "34f8f88e-3537-4e62-ae87-b12ec3812f62" (UID: "34f8f88e-3537-4e62-ae87-b12ec3812f62"). InnerVolumeSpecName "kube-api-access-p6lkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:36.965717 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:36.965698 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" Oct 30 20:00:37.050469 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.050443 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e60baa7a-9536-4dd5-b59c-d042cd41b798-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60baa7a-9536-4dd5-b59c-d042cd41b798-plugins-config-volume\") pod \"e60baa7a-9536-4dd5-b59c-d042cd41b798\" (UID: \"e60baa7a-9536-4dd5-b59c-d042cd41b798\") " Oct 30 20:00:37.050596 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.050483 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60baa7a-9536-4dd5-b59c-d042cd41b798-kube-api-access-4vvcz\" (UniqueName: \"kubernetes.io/projected/e60baa7a-9536-4dd5-b59c-d042cd41b798-kube-api-access-4vvcz\") pod \"e60baa7a-9536-4dd5-b59c-d042cd41b798\" (UID: \"e60baa7a-9536-4dd5-b59c-d042cd41b798\") " Oct 30 20:00:37.050733 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.050721 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6lkf\" (UniqueName: \"kubernetes.io/projected/34f8f88e-3537-4e62-ae87-b12ec3812f62-kube-api-access-p6lkf\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:37.050812 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.050793 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60baa7a-9536-4dd5-b59c-d042cd41b798-plugins-config-volume" pod "e60baa7a-9536-4dd5-b59c-d042cd41b798" (UID: "e60baa7a-9536-4dd5-b59c-d042cd41b798"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:37.052507 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.052487 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60baa7a-9536-4dd5-b59c-d042cd41b798-kube-api-access-4vvcz" pod "e60baa7a-9536-4dd5-b59c-d042cd41b798" (UID: "e60baa7a-9536-4dd5-b59c-d042cd41b798"). InnerVolumeSpecName "kube-api-access-4vvcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:37.152263 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.152237 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60baa7a-9536-4dd5-b59c-d042cd41b798-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:37.152263 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.152262 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vvcz\" (UniqueName: \"kubernetes.io/projected/e60baa7a-9536-4dd5-b59c-d042cd41b798-kube-api-access-4vvcz\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:00:37.632458 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.632432 4433 generic.go:356] "Generic (PLEG): container finished" podID="34f8f88e-3537-4e62-ae87-b12ec3812f62" containerID="b03cdc10b9b3c337f6ee33f18f184de3843bff71f25c515def64405888e84828" exitCode=0 Oct 30 20:00:37.632603 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.632484 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" Oct 30 20:00:37.632603 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.632519 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" event={"ID":"34f8f88e-3537-4e62-ae87-b12ec3812f62","Type":"ContainerDied","Data":"b03cdc10b9b3c337f6ee33f18f184de3843bff71f25c515def64405888e84828"} Oct 30 20:00:37.632603 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.632570 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd" event={"ID":"34f8f88e-3537-4e62-ae87-b12ec3812f62","Type":"ContainerDied","Data":"682e8522b2c2c24988a6bd20a1ff0641d1130db10dea33e4ce205658f432afd7"} Oct 30 20:00:37.632603 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.632592 4433 scope.go:122] "RemoveContainer" containerID="b03cdc10b9b3c337f6ee33f18f184de3843bff71f25c515def64405888e84828" Oct 30 20:00:37.634270 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.634238 4433 generic.go:356] "Generic (PLEG): container finished" podID="e60baa7a-9536-4dd5-b59c-d042cd41b798" containerID="7aa07c6e89eafcb90edfce328f8c892e08d8552e831a34b78f005fb76b0af478" exitCode=0 Oct 30 20:00:37.634387 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.634274 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" Oct 30 20:00:37.634387 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.634286 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" event={"ID":"e60baa7a-9536-4dd5-b59c-d042cd41b798","Type":"ContainerDied","Data":"7aa07c6e89eafcb90edfce328f8c892e08d8552e831a34b78f005fb76b0af478"} Oct 30 20:00:37.634387 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.634327 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh" event={"ID":"e60baa7a-9536-4dd5-b59c-d042cd41b798","Type":"ContainerDied","Data":"10e3e7e4dcf2a133e9d2c59acf7f8038ab093bfaf114b279cc4a83fa1ca2f698"} Oct 30 20:00:37.642536 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.642517 4433 scope.go:122] "RemoveContainer" containerID="b03cdc10b9b3c337f6ee33f18f184de3843bff71f25c515def64405888e84828" Oct 30 20:00:37.642807 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:37.642785 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03cdc10b9b3c337f6ee33f18f184de3843bff71f25c515def64405888e84828\": container with ID starting with b03cdc10b9b3c337f6ee33f18f184de3843bff71f25c515def64405888e84828 not found: ID does not exist" containerID="b03cdc10b9b3c337f6ee33f18f184de3843bff71f25c515def64405888e84828" Oct 30 20:00:37.642853 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.642819 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03cdc10b9b3c337f6ee33f18f184de3843bff71f25c515def64405888e84828"} err="failed to get container status \"b03cdc10b9b3c337f6ee33f18f184de3843bff71f25c515def64405888e84828\": rpc error: code = NotFound desc = could not find container \"b03cdc10b9b3c337f6ee33f18f184de3843bff71f25c515def64405888e84828\": container with ID starting with b03cdc10b9b3c337f6ee33f18f184de3843bff71f25c515def64405888e84828 not found: ID does not exist" Oct 30 20:00:37.642853 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.642837 4433 scope.go:122] "RemoveContainer" containerID="7aa07c6e89eafcb90edfce328f8c892e08d8552e831a34b78f005fb76b0af478" Oct 30 20:00:37.651141 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.651124 4433 scope.go:122] "RemoveContainer" containerID="7aa07c6e89eafcb90edfce328f8c892e08d8552e831a34b78f005fb76b0af478" Oct 30 20:00:37.651375 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:37.651349 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa07c6e89eafcb90edfce328f8c892e08d8552e831a34b78f005fb76b0af478\": container with ID starting with 7aa07c6e89eafcb90edfce328f8c892e08d8552e831a34b78f005fb76b0af478 not found: ID does not exist" containerID="7aa07c6e89eafcb90edfce328f8c892e08d8552e831a34b78f005fb76b0af478" Oct 30 20:00:37.651424 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.651376 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa07c6e89eafcb90edfce328f8c892e08d8552e831a34b78f005fb76b0af478"} err="failed to get container status \"7aa07c6e89eafcb90edfce328f8c892e08d8552e831a34b78f005fb76b0af478\": rpc error: code = NotFound desc = could not find container \"7aa07c6e89eafcb90edfce328f8c892e08d8552e831a34b78f005fb76b0af478\": container with ID starting with 7aa07c6e89eafcb90edfce328f8c892e08d8552e831a34b78f005fb76b0af478 not found: ID does not exist" Oct 30 20:00:37.798021 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.797992 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd"] Oct 30 20:00:37.846852 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.846786 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-686689-fc7484d58-9t7nd"] Oct 30 20:00:37.915770 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.915739 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh"] Oct 30 20:00:37.919373 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:37.919351 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-686689-59f598f9bc-p2jxh"] Oct 30 20:00:39.213270 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:39.213231 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f8f88e-3537-4e62-ae87-b12ec3812f62" path="/var/lib/kubelet/pods/34f8f88e-3537-4e62-ae87-b12ec3812f62/volumes" Oct 30 20:00:39.213787 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:39.213767 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e60baa7a-9536-4dd5-b59c-d042cd41b798" path="/var/lib/kubelet/pods/e60baa7a-9536-4dd5-b59c-d042cd41b798/volumes" Oct 30 20:00:41.372045 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:41.372009 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:43.028522 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:43.028484 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:43.955463 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:43.955434 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:44.233650 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:44.233569 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:44.517412 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:44.517334 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:44.766364 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:44.766307 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:45.317148 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:45.317114 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:45.623262 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:45.623182 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:46.584869 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:46.584830 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:47.145464 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:47.145429 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:47.479089 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:47.475898 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:48.314146 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:48.314113 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:48.646358 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:48.646269 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:48.909528 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:48.909442 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:49.336770 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:49.336740 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:50.731328 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:50.731279 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:51.992293 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:51.992260 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:52.331437 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:52.331407 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:53.367608 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:00:53.367568 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:00:53.929813 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:53.929764 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:56.102434 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:56.102401 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:56.389054 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:56.388965 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:56.665000 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:56.664911 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:56.975017 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:56.974917 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:59.023174 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:59.023139 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:59.269336 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:00:59.269300 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:01.810034 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:01.810002 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k"] Oct 30 20:01:01.992080 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:01.992038 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:03.557339 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:03.557305 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:05.241157 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:05.241122 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:05.503457 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:05.503343 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:05.784008 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:05.783905 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:06.145728 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:06.145694 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:09.350963 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:09.350928 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:09.598580 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:09.598547 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:10.736614 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:10.736582 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:12.469237 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:12.469197 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:12.784907 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:12.784829 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:13.094557 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:13.094524 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:15.812131 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:15.812078 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:16.830420 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:16.830383 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" containerID="cri-o://9f14f36cd46a72086207af110f0522bfdb9fca4fe26e3bf65ed978e3dfc3f5fe" gracePeriod=45 Oct 30 20:01:17.148005 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.147949 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 20:01:17.302498 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.302462 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-dshm\") pod \"88b9f088-a105-4af4-82ad-3efdf214a5e2\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " Oct 30 20:01:17.302690 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.302548 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88b9f088-a105-4af4-82ad-3efdf214a5e2-tls-certs\" (UniqueName: \"kubernetes.io/secret/88b9f088-a105-4af4-82ad-3efdf214a5e2-tls-certs\") pod \"88b9f088-a105-4af4-82ad-3efdf214a5e2\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " Oct 30 20:01:17.302690 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.302588 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-home\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-home\") pod \"88b9f088-a105-4af4-82ad-3efdf214a5e2\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " Oct 30 20:01:17.302690 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.302671 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-tmp-dir\") pod \"88b9f088-a105-4af4-82ad-3efdf214a5e2\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " Oct 30 20:01:17.302889 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.302699 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-model-cache\") pod \"88b9f088-a105-4af4-82ad-3efdf214a5e2\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " Oct 30 20:01:17.302889 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.302772 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b9f088-a105-4af4-82ad-3efdf214a5e2-kube-api-access-bh5fl\" (UniqueName: \"kubernetes.io/projected/88b9f088-a105-4af4-82ad-3efdf214a5e2-kube-api-access-bh5fl\") pod \"88b9f088-a105-4af4-82ad-3efdf214a5e2\" (UID: \"88b9f088-a105-4af4-82ad-3efdf214a5e2\") " Oct 30 20:01:17.303286 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.303240 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-home" pod "88b9f088-a105-4af4-82ad-3efdf214a5e2" (UID: "88b9f088-a105-4af4-82ad-3efdf214a5e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:17.304046 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.304023 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-model-cache" pod "88b9f088-a105-4af4-82ad-3efdf214a5e2" (UID: "88b9f088-a105-4af4-82ad-3efdf214a5e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:17.304186 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.304039 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-tmp-dir" pod "88b9f088-a105-4af4-82ad-3efdf214a5e2" (UID: "88b9f088-a105-4af4-82ad-3efdf214a5e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:17.307245 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.307214 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b9f088-a105-4af4-82ad-3efdf214a5e2-tls-certs" pod "88b9f088-a105-4af4-82ad-3efdf214a5e2" (UID: "88b9f088-a105-4af4-82ad-3efdf214a5e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:17.308685 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.308647 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-dshm" pod "88b9f088-a105-4af4-82ad-3efdf214a5e2" (UID: "88b9f088-a105-4af4-82ad-3efdf214a5e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:17.313875 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.313847 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b9f088-a105-4af4-82ad-3efdf214a5e2-kube-api-access-bh5fl" pod "88b9f088-a105-4af4-82ad-3efdf214a5e2" (UID: "88b9f088-a105-4af4-82ad-3efdf214a5e2"). InnerVolumeSpecName "kube-api-access-bh5fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:17.404791 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.404708 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-dshm\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.404791 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.404735 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b9f088-a105-4af4-82ad-3efdf214a5e2-tls-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.404791 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.404744 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-home\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.404791 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.404751 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-tmp-dir\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.404791 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.404760 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88b9f088-a105-4af4-82ad-3efdf214a5e2-model-cache\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.404791 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.404770 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh5fl\" (UniqueName: \"kubernetes.io/projected/88b9f088-a105-4af4-82ad-3efdf214a5e2-kube-api-access-bh5fl\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.904283 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.904241 4433 generic.go:356] "Generic (PLEG): container finished" podID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerID="9f14f36cd46a72086207af110f0522bfdb9fca4fe26e3bf65ed978e3dfc3f5fe" exitCode=0 Oct 30 20:01:17.904733 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.904296 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" Oct 30 20:01:17.904733 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.904324 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" event={"ID":"88b9f088-a105-4af4-82ad-3efdf214a5e2","Type":"ContainerDied","Data":"9f14f36cd46a72086207af110f0522bfdb9fca4fe26e3bf65ed978e3dfc3f5fe"} Oct 30 20:01:17.904733 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.904369 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k" event={"ID":"88b9f088-a105-4af4-82ad-3efdf214a5e2","Type":"ContainerDied","Data":"d7da3d506c29c2cc9fe67b5deb1571ebc54187866300e359e59a4f940a375232"} Oct 30 20:01:17.904733 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.904388 4433 scope.go:122] "RemoveContainer" containerID="9f14f36cd46a72086207af110f0522bfdb9fca4fe26e3bf65ed978e3dfc3f5fe" Oct 30 20:01:17.914841 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.914824 4433 scope.go:122] "RemoveContainer" containerID="9f14f36cd46a72086207af110f0522bfdb9fca4fe26e3bf65ed978e3dfc3f5fe" Oct 30 20:01:17.915144 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:01:17.915124 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f14f36cd46a72086207af110f0522bfdb9fca4fe26e3bf65ed978e3dfc3f5fe\": container with ID starting with 9f14f36cd46a72086207af110f0522bfdb9fca4fe26e3bf65ed978e3dfc3f5fe not found: ID does not exist" containerID="9f14f36cd46a72086207af110f0522bfdb9fca4fe26e3bf65ed978e3dfc3f5fe" Oct 30 20:01:17.915215 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:17.915154 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f14f36cd46a72086207af110f0522bfdb9fca4fe26e3bf65ed978e3dfc3f5fe"} err="failed to get container status \"9f14f36cd46a72086207af110f0522bfdb9fca4fe26e3bf65ed978e3dfc3f5fe\": rpc error: code = NotFound desc = could not find container \"9f14f36cd46a72086207af110f0522bfdb9fca4fe26e3bf65ed978e3dfc3f5fe\": container with ID starting with 9f14f36cd46a72086207af110f0522bfdb9fca4fe26e3bf65ed978e3dfc3f5fe not found: ID does not exist" Oct 30 20:01:18.072659 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:18.072618 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k"] Oct 30 20:01:18.120319 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:18.120288 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/distinct-w0-2d9861-kserve-5976675cd7-6qd7k"] Oct 30 20:01:19.213947 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:19.213914 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" path="/var/lib/kubelet/pods/88b9f088-a105-4af4-82ad-3efdf214a5e2/volumes" Oct 30 20:01:21.292620 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:21.292587 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:23.365936 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:01:23.365898 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:01:25.498585 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:25.498551 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:25.747758 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:25.747721 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:26.612175 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:26.612138 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj"] Oct 30 20:01:26.614662 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:26.614611 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" podUID="aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0" containerName="maas-api" containerID="cri-o://9a9855312bc9447b2d0bba4c54b416af520d5a6ba2552e5036c66f0dca873df8" gracePeriod=30 Oct 30 20:01:26.653333 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:26.649380 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:26.879572 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:26.879467 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:26.933112 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:26.933083 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 20:01:26.958801 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:26.958766 4433 generic.go:356] "Generic (PLEG): container finished" podID="aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0" containerID="9a9855312bc9447b2d0bba4c54b416af520d5a6ba2552e5036c66f0dca873df8" exitCode=0 Oct 30 20:01:26.958966 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:26.958813 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" Oct 30 20:01:26.958966 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:26.958843 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" event={"ID":"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0","Type":"ContainerDied","Data":"9a9855312bc9447b2d0bba4c54b416af520d5a6ba2552e5036c66f0dca873df8"} Oct 30 20:01:26.958966 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:26.958899 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj" event={"ID":"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0","Type":"ContainerDied","Data":"605ef0eda04283c854413a70c952d433cf0ec17ed0e4d2af5d0df35d57695f2f"} Oct 30 20:01:26.958966 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:26.958920 4433 scope.go:122] "RemoveContainer" containerID="9a9855312bc9447b2d0bba4c54b416af520d5a6ba2552e5036c66f0dca873df8" Oct 30 20:01:26.991108 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:26.991078 4433 scope.go:122] "RemoveContainer" containerID="9a9855312bc9447b2d0bba4c54b416af520d5a6ba2552e5036c66f0dca873df8" Oct 30 20:01:26.991828 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:01:26.991785 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a9855312bc9447b2d0bba4c54b416af520d5a6ba2552e5036c66f0dca873df8\": container with ID starting with 9a9855312bc9447b2d0bba4c54b416af520d5a6ba2552e5036c66f0dca873df8 not found: ID does not exist" containerID="9a9855312bc9447b2d0bba4c54b416af520d5a6ba2552e5036c66f0dca873df8" Oct 30 20:01:26.991948 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:26.991840 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a9855312bc9447b2d0bba4c54b416af520d5a6ba2552e5036c66f0dca873df8"} err="failed to get container status \"9a9855312bc9447b2d0bba4c54b416af520d5a6ba2552e5036c66f0dca873df8\": rpc error: code = NotFound desc = could not find container \"9a9855312bc9447b2d0bba4c54b416af520d5a6ba2552e5036c66f0dca873df8\": container with ID starting with 9a9855312bc9447b2d0bba4c54b416af520d5a6ba2552e5036c66f0dca873df8 not found: ID does not exist" Oct 30 20:01:27.042766 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:27.042735 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-kube-api-access-fxjk7\" (UniqueName: \"kubernetes.io/projected/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-kube-api-access-fxjk7\") pod \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " Oct 30 20:01:27.042956 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:27.042892 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-metrics-certs\") pod \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " Oct 30 20:01:27.042956 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:27.042916 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-openshift-service-ca\") pod \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " Oct 30 20:01:27.043124 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:27.043042 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-maas-api-tls\") pod \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\" (UID: \"aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0\") " Oct 30 20:01:27.043350 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:27.043324 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-openshift-service-ca" pod "aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0" (UID: "aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:27.043532 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:27.043514 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-openshift-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:27.045009 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:27.044966 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-maas-api-tls" pod "aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0" (UID: "aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:27.045101 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:27.045028 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-kube-api-access-fxjk7" pod "aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0" (UID: "aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0"). InnerVolumeSpecName "kube-api-access-fxjk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:27.045101 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:27.045092 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-metrics-certs" pod "aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0" (UID: "aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:27.144483 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:27.144401 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-metrics-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:27.144483 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:27.144433 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-maas-api-tls\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:27.144483 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:27.144444 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxjk7\" (UniqueName: \"kubernetes.io/projected/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0-kube-api-access-fxjk7\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:27.174159 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:27.172847 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:27.374463 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:27.374425 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj"] Oct 30 20:01:27.378549 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:27.378521 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f55cd-c695c9f8d-wshmj"] Oct 30 20:01:27.453130 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:27.449160 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:28.115335 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:28.115304 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:28.391217 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:28.391131 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:28.639450 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:28.639414 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:29.005254 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:29.005217 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:29.214621 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:29.214582 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0" path="/var/lib/kubelet/pods/aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0/volumes" Oct 30 20:01:29.274443 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:29.274356 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb"] Oct 30 20:01:29.274619 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:29.274589 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" podUID="1550c04a-cebb-456c-9a05-4eeddaefef8a" containerName="istio-proxy" containerID="cri-o://c25dd67b3fc028d75690d134d0eb567ff717e7cf448facf4d65aaccb41e5e3ee" gracePeriod=30 Oct 30 20:01:29.299219 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:29.299187 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv"] Oct 30 20:01:29.299407 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:29.299387 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" podUID="ff913782-d7e9-4110-a499-623fda834acd" containerName="istio-proxy" containerID="cri-o://14c0c02aaacf0e9d4108c0990c8304d2bee4aacb801f64b9b85abc28bd9950eb" gracePeriod=30 Oct 30 20:01:29.299708 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:29.299603 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:29.549165 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:29.549084 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:31.463802 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:31.463766 4433 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:01:31.464193 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:31.463827 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" podUID="1550c04a-cebb-456c-9a05-4eeddaefef8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:01:33.956113 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:33.956079 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:34.257888 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.257803 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:34.505015 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.504956 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:34.512140 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.512073 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj"] Oct 30 20:01:34.512659 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.512300 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" podUID="3af9891c-2337-4531-a662-1eb43724b440" containerName="istio-proxy" containerID="cri-o://4aa6244a56d4d2a5bbbafd45bb45bebfce9fd530567d772b12561b79ed53a8e8" gracePeriod=30 Oct 30 20:01:34.639414 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.639378 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 20:01:34.682593 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.682566 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 20:01:34.743676 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.743604 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1550c04a-cebb-456c-9a05-4eeddaefef8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1550c04a-cebb-456c-9a05-4eeddaefef8a-istiod-ca-cert\") pod \"1550c04a-cebb-456c-9a05-4eeddaefef8a\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " Oct 30 20:01:34.743676 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.743660 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-token\" (UniqueName: \"kubernetes.io/projected/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-token\") pod \"1550c04a-cebb-456c-9a05-4eeddaefef8a\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " Oct 30 20:01:34.743910 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.743711 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-workload-certs\") pod \"1550c04a-cebb-456c-9a05-4eeddaefef8a\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " Oct 30 20:01:34.743910 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.743749 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-envoy\") pod \"1550c04a-cebb-456c-9a05-4eeddaefef8a\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " Oct 30 20:01:34.743910 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.743772 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-data\") pod \"1550c04a-cebb-456c-9a05-4eeddaefef8a\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " Oct 30 20:01:34.743910 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.743802 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-podinfo\") pod \"1550c04a-cebb-456c-9a05-4eeddaefef8a\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " Oct 30 20:01:34.743910 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.743836 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-credential-socket\") pod \"1550c04a-cebb-456c-9a05-4eeddaefef8a\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " Oct 30 20:01:34.744587 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.744473 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1550c04a-cebb-456c-9a05-4eeddaefef8a-istiod-ca-cert" pod "1550c04a-cebb-456c-9a05-4eeddaefef8a" (UID: "1550c04a-cebb-456c-9a05-4eeddaefef8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:34.744696 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.744593 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-workload-socket" pod "1550c04a-cebb-456c-9a05-4eeddaefef8a" (UID: "1550c04a-cebb-456c-9a05-4eeddaefef8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:34.744696 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.744622 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-credential-socket" pod "1550c04a-cebb-456c-9a05-4eeddaefef8a" (UID: "1550c04a-cebb-456c-9a05-4eeddaefef8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:34.744696 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.744671 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-workload-certs" pod "1550c04a-cebb-456c-9a05-4eeddaefef8a" (UID: "1550c04a-cebb-456c-9a05-4eeddaefef8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:34.744853 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.744686 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-data" pod "1550c04a-cebb-456c-9a05-4eeddaefef8a" (UID: "1550c04a-cebb-456c-9a05-4eeddaefef8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:34.744853 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.744763 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-workload-socket\") pod \"1550c04a-cebb-456c-9a05-4eeddaefef8a\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " Oct 30 20:01:34.744953 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.744887 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1550c04a-cebb-456c-9a05-4eeddaefef8a-kube-api-access-hnxxx\" (UniqueName: \"kubernetes.io/projected/1550c04a-cebb-456c-9a05-4eeddaefef8a-kube-api-access-hnxxx\") pod \"1550c04a-cebb-456c-9a05-4eeddaefef8a\" (UID: \"1550c04a-cebb-456c-9a05-4eeddaefef8a\") " Oct 30 20:01:34.746648 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.746594 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.746648 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.746629 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.746648 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.746644 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.746846 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.746666 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.746846 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.746680 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1550c04a-cebb-456c-9a05-4eeddaefef8a-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.747033 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.746956 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-podinfo" pod "1550c04a-cebb-456c-9a05-4eeddaefef8a" (UID: "1550c04a-cebb-456c-9a05-4eeddaefef8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:01:34.747456 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.747422 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-envoy" pod "1550c04a-cebb-456c-9a05-4eeddaefef8a" (UID: "1550c04a-cebb-456c-9a05-4eeddaefef8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:34.747865 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.747837 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:34.755379 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.755353 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-token" pod "1550c04a-cebb-456c-9a05-4eeddaefef8a" (UID: "1550c04a-cebb-456c-9a05-4eeddaefef8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:34.755562 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.755549 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1550c04a-cebb-456c-9a05-4eeddaefef8a-kube-api-access-hnxxx" pod "1550c04a-cebb-456c-9a05-4eeddaefef8a" (UID: "1550c04a-cebb-456c-9a05-4eeddaefef8a"). InnerVolumeSpecName "kube-api-access-hnxxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:34.847920 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.847886 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff913782-d7e9-4110-a499-623fda834acd-istio-token\" (UniqueName: \"kubernetes.io/projected/ff913782-d7e9-4110-a499-623fda834acd-istio-token\") pod \"ff913782-d7e9-4110-a499-623fda834acd\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " Oct 30 20:01:34.848129 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.847948 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff913782-d7e9-4110-a499-623fda834acd-kube-api-access-8ddkx\" (UniqueName: \"kubernetes.io/projected/ff913782-d7e9-4110-a499-623fda834acd-kube-api-access-8ddkx\") pod \"ff913782-d7e9-4110-a499-623fda834acd\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " Oct 30 20:01:34.848129 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.847975 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff913782-d7e9-4110-a499-623fda834acd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff913782-d7e9-4110-a499-623fda834acd-istio-podinfo\") pod \"ff913782-d7e9-4110-a499-623fda834acd\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " Oct 30 20:01:34.848129 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848012 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff913782-d7e9-4110-a499-623fda834acd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff913782-d7e9-4110-a499-623fda834acd-istiod-ca-cert\") pod \"ff913782-d7e9-4110-a499-623fda834acd\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " Oct 30 20:01:34.848129 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848071 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-istio-data\") pod \"ff913782-d7e9-4110-a499-623fda834acd\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " Oct 30 20:01:34.848129 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848123 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-istio-envoy\") pod \"ff913782-d7e9-4110-a499-623fda834acd\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " Oct 30 20:01:34.848408 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848169 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-workload-socket\") pod \"ff913782-d7e9-4110-a499-623fda834acd\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " Oct 30 20:01:34.848408 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848195 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-workload-certs\") pod \"ff913782-d7e9-4110-a499-623fda834acd\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " Oct 30 20:01:34.848408 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848227 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-credential-socket\") pod \"ff913782-d7e9-4110-a499-623fda834acd\" (UID: \"ff913782-d7e9-4110-a499-623fda834acd\") " Oct 30 20:01:34.848575 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848459 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-istio-data" pod "ff913782-d7e9-4110-a499-623fda834acd" (UID: "ff913782-d7e9-4110-a499-623fda834acd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:34.848575 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848494 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff913782-d7e9-4110-a499-623fda834acd-istiod-ca-cert" pod "ff913782-d7e9-4110-a499-623fda834acd" (UID: "ff913782-d7e9-4110-a499-623fda834acd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:34.848744 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848720 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-workload-socket" pod "ff913782-d7e9-4110-a499-623fda834acd" (UID: "ff913782-d7e9-4110-a499-623fda834acd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:34.848802 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848739 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-workload-certs" pod "ff913782-d7e9-4110-a499-623fda834acd" (UID: "ff913782-d7e9-4110-a499-623fda834acd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:34.848862 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848833 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff913782-d7e9-4110-a499-623fda834acd-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.848902 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848858 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.848902 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848873 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.848902 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848888 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.849037 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848903 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.849037 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848915 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.849037 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848929 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnxxx\" (UniqueName: \"kubernetes.io/projected/1550c04a-cebb-456c-9a05-4eeddaefef8a-kube-api-access-hnxxx\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.849037 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.848943 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1550c04a-cebb-456c-9a05-4eeddaefef8a-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.849349 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.849327 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-credential-socket" pod "ff913782-d7e9-4110-a499-623fda834acd" (UID: "ff913782-d7e9-4110-a499-623fda834acd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:34.850224 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.850199 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff913782-d7e9-4110-a499-623fda834acd-istio-token" pod "ff913782-d7e9-4110-a499-623fda834acd" (UID: "ff913782-d7e9-4110-a499-623fda834acd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:34.850315 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.850301 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-istio-envoy" pod "ff913782-d7e9-4110-a499-623fda834acd" (UID: "ff913782-d7e9-4110-a499-623fda834acd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:34.850576 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.850552 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff913782-d7e9-4110-a499-623fda834acd-istio-podinfo" pod "ff913782-d7e9-4110-a499-623fda834acd" (UID: "ff913782-d7e9-4110-a499-623fda834acd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:01:34.850673 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.850574 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff913782-d7e9-4110-a499-623fda834acd-kube-api-access-8ddkx" pod "ff913782-d7e9-4110-a499-623fda834acd" (UID: "ff913782-d7e9-4110-a499-623fda834acd"). InnerVolumeSpecName "kube-api-access-8ddkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:34.950188 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.950150 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.950188 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.950190 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff913782-d7e9-4110-a499-623fda834acd-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.950391 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.950204 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff913782-d7e9-4110-a499-623fda834acd-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.950391 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.950219 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ddkx\" (UniqueName: \"kubernetes.io/projected/ff913782-d7e9-4110-a499-623fda834acd-kube-api-access-8ddkx\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.950391 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:34.950234 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff913782-d7e9-4110-a499-623fda834acd-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:35.008142 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.008109 4433 generic.go:356] "Generic (PLEG): container finished" podID="1550c04a-cebb-456c-9a05-4eeddaefef8a" containerID="c25dd67b3fc028d75690d134d0eb567ff717e7cf448facf4d65aaccb41e5e3ee" exitCode=0 Oct 30 20:01:35.008620 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.008172 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" Oct 30 20:01:35.008620 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.008198 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" event={"ID":"1550c04a-cebb-456c-9a05-4eeddaefef8a","Type":"ContainerDied","Data":"c25dd67b3fc028d75690d134d0eb567ff717e7cf448facf4d65aaccb41e5e3ee"} Oct 30 20:01:35.008620 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.008282 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb" event={"ID":"1550c04a-cebb-456c-9a05-4eeddaefef8a","Type":"ContainerDied","Data":"03186d34b1561d58961a1f1f236ddc0fefbe4b62a94d015782f928f380e17ec3"} Oct 30 20:01:35.008620 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.008310 4433 scope.go:122] "RemoveContainer" containerID="c25dd67b3fc028d75690d134d0eb567ff717e7cf448facf4d65aaccb41e5e3ee" Oct 30 20:01:35.010053 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.010027 4433 generic.go:356] "Generic (PLEG): container finished" podID="ff913782-d7e9-4110-a499-623fda834acd" containerID="14c0c02aaacf0e9d4108c0990c8304d2bee4aacb801f64b9b85abc28bd9950eb" exitCode=0 Oct 30 20:01:35.010152 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.010064 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" event={"ID":"ff913782-d7e9-4110-a499-623fda834acd","Type":"ContainerDied","Data":"14c0c02aaacf0e9d4108c0990c8304d2bee4aacb801f64b9b85abc28bd9950eb"} Oct 30 20:01:35.010152 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.010109 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" event={"ID":"ff913782-d7e9-4110-a499-623fda834acd","Type":"ContainerDied","Data":"bb486eca0fa5940cb3680d169bd8b23d0f220fa522b42a538b50b8fde7217dc6"} Oct 30 20:01:35.010152 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.010108 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv" Oct 30 20:01:35.019841 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.019810 4433 scope.go:122] "RemoveContainer" containerID="c25dd67b3fc028d75690d134d0eb567ff717e7cf448facf4d65aaccb41e5e3ee" Oct 30 20:01:35.020195 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:01:35.020159 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c25dd67b3fc028d75690d134d0eb567ff717e7cf448facf4d65aaccb41e5e3ee\": container with ID starting with c25dd67b3fc028d75690d134d0eb567ff717e7cf448facf4d65aaccb41e5e3ee not found: ID does not exist" containerID="c25dd67b3fc028d75690d134d0eb567ff717e7cf448facf4d65aaccb41e5e3ee" Oct 30 20:01:35.020301 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.020194 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25dd67b3fc028d75690d134d0eb567ff717e7cf448facf4d65aaccb41e5e3ee"} err="failed to get container status \"c25dd67b3fc028d75690d134d0eb567ff717e7cf448facf4d65aaccb41e5e3ee\": rpc error: code = NotFound desc = could not find container \"c25dd67b3fc028d75690d134d0eb567ff717e7cf448facf4d65aaccb41e5e3ee\": container with ID starting with c25dd67b3fc028d75690d134d0eb567ff717e7cf448facf4d65aaccb41e5e3ee not found: ID does not exist" Oct 30 20:01:35.020301 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.020213 4433 scope.go:122] "RemoveContainer" containerID="14c0c02aaacf0e9d4108c0990c8304d2bee4aacb801f64b9b85abc28bd9950eb" Oct 30 20:01:35.029275 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.029256 4433 scope.go:122] "RemoveContainer" containerID="b838f891e6cb72d77fdf752fe5f534f2e59d77bb0f6a1e138db7b8e64c0a96e6" Oct 30 20:01:35.037390 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.037372 4433 scope.go:122] "RemoveContainer" containerID="14c0c02aaacf0e9d4108c0990c8304d2bee4aacb801f64b9b85abc28bd9950eb" Oct 30 20:01:35.037659 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:01:35.037640 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c0c02aaacf0e9d4108c0990c8304d2bee4aacb801f64b9b85abc28bd9950eb\": container with ID starting with 14c0c02aaacf0e9d4108c0990c8304d2bee4aacb801f64b9b85abc28bd9950eb not found: ID does not exist" containerID="14c0c02aaacf0e9d4108c0990c8304d2bee4aacb801f64b9b85abc28bd9950eb" Oct 30 20:01:35.037739 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.037670 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c0c02aaacf0e9d4108c0990c8304d2bee4aacb801f64b9b85abc28bd9950eb"} err="failed to get container status \"14c0c02aaacf0e9d4108c0990c8304d2bee4aacb801f64b9b85abc28bd9950eb\": rpc error: code = NotFound desc = could not find container \"14c0c02aaacf0e9d4108c0990c8304d2bee4aacb801f64b9b85abc28bd9950eb\": container with ID starting with 14c0c02aaacf0e9d4108c0990c8304d2bee4aacb801f64b9b85abc28bd9950eb not found: ID does not exist" Oct 30 20:01:35.037739 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.037694 4433 scope.go:122] "RemoveContainer" containerID="b838f891e6cb72d77fdf752fe5f534f2e59d77bb0f6a1e138db7b8e64c0a96e6" Oct 30 20:01:35.037936 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:01:35.037918 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b838f891e6cb72d77fdf752fe5f534f2e59d77bb0f6a1e138db7b8e64c0a96e6\": container with ID starting with b838f891e6cb72d77fdf752fe5f534f2e59d77bb0f6a1e138db7b8e64c0a96e6 not found: ID does not exist" containerID="b838f891e6cb72d77fdf752fe5f534f2e59d77bb0f6a1e138db7b8e64c0a96e6" Oct 30 20:01:35.037998 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.037943 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b838f891e6cb72d77fdf752fe5f534f2e59d77bb0f6a1e138db7b8e64c0a96e6"} err="failed to get container status \"b838f891e6cb72d77fdf752fe5f534f2e59d77bb0f6a1e138db7b8e64c0a96e6\": rpc error: code = NotFound desc = could not find container \"b838f891e6cb72d77fdf752fe5f534f2e59d77bb0f6a1e138db7b8e64c0a96e6\": container with ID starting with b838f891e6cb72d77fdf752fe5f534f2e59d77bb0f6a1e138db7b8e64c0a96e6 not found: ID does not exist" Oct 30 20:01:35.177168 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.177088 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb"] Oct 30 20:01:35.187354 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.187328 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-xdltb"] Oct 30 20:01:35.213834 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.213810 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1550c04a-cebb-456c-9a05-4eeddaefef8a" path="/var/lib/kubelet/pods/1550c04a-cebb-456c-9a05-4eeddaefef8a/volumes" Oct 30 20:01:35.312943 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.312916 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv"] Oct 30 20:01:35.320590 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:35.320563 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-jzcdv"] Oct 30 20:01:37.213415 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:37.213382 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff913782-d7e9-4110-a499-623fda834acd" path="/var/lib/kubelet/pods/ff913782-d7e9-4110-a499-623fda834acd/volumes" Oct 30 20:01:38.920081 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:38.920049 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:39.551535 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.551508 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:39.763921 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.763895 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 20:01:39.906208 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.906179 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af9891c-2337-4531-a662-1eb43724b440-istio-token\" (UniqueName: \"kubernetes.io/projected/3af9891c-2337-4531-a662-1eb43724b440-istio-token\") pod \"3af9891c-2337-4531-a662-1eb43724b440\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " Oct 30 20:01:39.906358 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.906248 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-workload-certs\") pod \"3af9891c-2337-4531-a662-1eb43724b440\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " Oct 30 20:01:39.906358 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.906275 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-istio-data\") pod \"3af9891c-2337-4531-a662-1eb43724b440\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " Oct 30 20:01:39.906358 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.906314 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af9891c-2337-4531-a662-1eb43724b440-kube-api-access-jk2gp\" (UniqueName: \"kubernetes.io/projected/3af9891c-2337-4531-a662-1eb43724b440-kube-api-access-jk2gp\") pod \"3af9891c-2337-4531-a662-1eb43724b440\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " Oct 30 20:01:39.906545 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.906391 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-credential-socket\") pod \"3af9891c-2337-4531-a662-1eb43724b440\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " Oct 30 20:01:39.906545 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.906423 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3af9891c-2337-4531-a662-1eb43724b440-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af9891c-2337-4531-a662-1eb43724b440-istio-podinfo\") pod \"3af9891c-2337-4531-a662-1eb43724b440\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " Oct 30 20:01:39.906545 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.906465 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-workload-socket\") pod \"3af9891c-2337-4531-a662-1eb43724b440\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " Oct 30 20:01:39.906545 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.906486 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3af9891c-2337-4531-a662-1eb43724b440-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af9891c-2337-4531-a662-1eb43724b440-istiod-ca-cert\") pod \"3af9891c-2337-4531-a662-1eb43724b440\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " Oct 30 20:01:39.906545 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.906528 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-istio-envoy\") pod \"3af9891c-2337-4531-a662-1eb43724b440\" (UID: \"3af9891c-2337-4531-a662-1eb43724b440\") " Oct 30 20:01:39.906867 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.906592 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-istio-data" pod "3af9891c-2337-4531-a662-1eb43724b440" (UID: "3af9891c-2337-4531-a662-1eb43724b440"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:39.906867 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.906580 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-workload-certs" pod "3af9891c-2337-4531-a662-1eb43724b440" (UID: "3af9891c-2337-4531-a662-1eb43724b440"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:39.906867 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.906647 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-credential-socket" pod "3af9891c-2337-4531-a662-1eb43724b440" (UID: "3af9891c-2337-4531-a662-1eb43724b440"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:39.906867 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.906788 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-workload-socket" pod "3af9891c-2337-4531-a662-1eb43724b440" (UID: "3af9891c-2337-4531-a662-1eb43724b440"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:39.907123 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.906919 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af9891c-2337-4531-a662-1eb43724b440-istiod-ca-cert" pod "3af9891c-2337-4531-a662-1eb43724b440" (UID: "3af9891c-2337-4531-a662-1eb43724b440"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:39.907123 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.907032 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:39.907123 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.907047 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:39.907123 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.907064 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:39.907123 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.907076 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:39.908517 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.908494 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af9891c-2337-4531-a662-1eb43724b440-istio-token" pod "3af9891c-2337-4531-a662-1eb43724b440" (UID: "3af9891c-2337-4531-a662-1eb43724b440"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:39.908673 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.908650 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af9891c-2337-4531-a662-1eb43724b440-kube-api-access-jk2gp" pod "3af9891c-2337-4531-a662-1eb43724b440" (UID: "3af9891c-2337-4531-a662-1eb43724b440"). InnerVolumeSpecName "kube-api-access-jk2gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:39.908918 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.908896 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3af9891c-2337-4531-a662-1eb43724b440-istio-podinfo" pod "3af9891c-2337-4531-a662-1eb43724b440" (UID: "3af9891c-2337-4531-a662-1eb43724b440"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:01:39.908918 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:39.908902 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-istio-envoy" pod "3af9891c-2337-4531-a662-1eb43724b440" (UID: "3af9891c-2337-4531-a662-1eb43724b440"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:40.008390 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.008358 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk2gp\" (UniqueName: \"kubernetes.io/projected/3af9891c-2337-4531-a662-1eb43724b440-kube-api-access-jk2gp\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:40.008390 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.008384 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af9891c-2337-4531-a662-1eb43724b440-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:40.008390 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.008394 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af9891c-2337-4531-a662-1eb43724b440-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:40.008798 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.008402 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af9891c-2337-4531-a662-1eb43724b440-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:40.008798 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.008410 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af9891c-2337-4531-a662-1eb43724b440-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:40.041738 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.041704 4433 generic.go:356] "Generic (PLEG): container finished" podID="3af9891c-2337-4531-a662-1eb43724b440" containerID="4aa6244a56d4d2a5bbbafd45bb45bebfce9fd530567d772b12561b79ed53a8e8" exitCode=0 Oct 30 20:01:40.041862 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.041755 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" Oct 30 20:01:40.041862 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.041790 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" event={"ID":"3af9891c-2337-4531-a662-1eb43724b440","Type":"ContainerDied","Data":"4aa6244a56d4d2a5bbbafd45bb45bebfce9fd530567d772b12561b79ed53a8e8"} Oct 30 20:01:40.041862 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.041846 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj" event={"ID":"3af9891c-2337-4531-a662-1eb43724b440","Type":"ContainerDied","Data":"786e75557a435db71d837f1fff74a946467fb4b00057522280210d44f69621e2"} Oct 30 20:01:40.042054 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.041875 4433 scope.go:122] "RemoveContainer" containerID="4aa6244a56d4d2a5bbbafd45bb45bebfce9fd530567d772b12561b79ed53a8e8" Oct 30 20:01:40.052372 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.052355 4433 scope.go:122] "RemoveContainer" containerID="e65334778340ec7e024b969ec770140dc3783c3c5a2426d2cd1b1e42113c274f" Oct 30 20:01:40.060207 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.060184 4433 scope.go:122] "RemoveContainer" containerID="4aa6244a56d4d2a5bbbafd45bb45bebfce9fd530567d772b12561b79ed53a8e8" Oct 30 20:01:40.060461 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:01:40.060440 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa6244a56d4d2a5bbbafd45bb45bebfce9fd530567d772b12561b79ed53a8e8\": container with ID starting with 4aa6244a56d4d2a5bbbafd45bb45bebfce9fd530567d772b12561b79ed53a8e8 not found: ID does not exist" containerID="4aa6244a56d4d2a5bbbafd45bb45bebfce9fd530567d772b12561b79ed53a8e8" Oct 30 20:01:40.060543 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.060467 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa6244a56d4d2a5bbbafd45bb45bebfce9fd530567d772b12561b79ed53a8e8"} err="failed to get container status \"4aa6244a56d4d2a5bbbafd45bb45bebfce9fd530567d772b12561b79ed53a8e8\": rpc error: code = NotFound desc = could not find container \"4aa6244a56d4d2a5bbbafd45bb45bebfce9fd530567d772b12561b79ed53a8e8\": container with ID starting with 4aa6244a56d4d2a5bbbafd45bb45bebfce9fd530567d772b12561b79ed53a8e8 not found: ID does not exist" Oct 30 20:01:40.060543 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.060486 4433 scope.go:122] "RemoveContainer" containerID="e65334778340ec7e024b969ec770140dc3783c3c5a2426d2cd1b1e42113c274f" Oct 30 20:01:40.060755 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:01:40.060736 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65334778340ec7e024b969ec770140dc3783c3c5a2426d2cd1b1e42113c274f\": container with ID starting with e65334778340ec7e024b969ec770140dc3783c3c5a2426d2cd1b1e42113c274f not found: ID does not exist" containerID="e65334778340ec7e024b969ec770140dc3783c3c5a2426d2cd1b1e42113c274f" Oct 30 20:01:40.060801 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.060760 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65334778340ec7e024b969ec770140dc3783c3c5a2426d2cd1b1e42113c274f"} err="failed to get container status \"e65334778340ec7e024b969ec770140dc3783c3c5a2426d2cd1b1e42113c274f\": rpc error: code = NotFound desc = could not find container \"e65334778340ec7e024b969ec770140dc3783c3c5a2426d2cd1b1e42113c274f\": container with ID starting with e65334778340ec7e024b969ec770140dc3783c3c5a2426d2cd1b1e42113c274f not found: ID does not exist" Oct 30 20:01:40.205017 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.204946 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj"] Oct 30 20:01:40.209149 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:40.209115 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-jgpqj"] Oct 30 20:01:41.213351 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:41.213319 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3af9891c-2337-4531-a662-1eb43724b440" path="/var/lib/kubelet/pods/3af9891c-2337-4531-a662-1eb43724b440/volumes" Oct 30 20:01:43.609141 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:43.609102 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:47.150960 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:47.150927 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:47.417143 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:47.417061 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx"] Oct 30 20:01:47.417297 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:47.417262 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" podUID="f2042f53-156d-42c2-9554-5fe559fbe56f" containerName="istio-proxy" containerID="cri-o://a7f3bcbd9ebc63079f1e74772b837350830b7c74bad2545cc97dbdaf8e120083" gracePeriod=30 Oct 30 20:01:47.461881 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:47.461849 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:47.824718 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:47.824682 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:48.166052 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:48.165947 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:48.431334 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:48.431251 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:48.733610 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:48.733520 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:49.013706 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:49.013615 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:50.262430 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:50.262395 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:51.095199 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:51.095155 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:52.690769 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.690713 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 20:01:52.755955 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.755922 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-workload-socket\") pod \"f2042f53-156d-42c2-9554-5fe559fbe56f\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " Oct 30 20:01:52.755955 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.755960 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-token\" (UniqueName: \"kubernetes.io/projected/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-token\") pod \"f2042f53-156d-42c2-9554-5fe559fbe56f\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " Oct 30 20:01:52.756204 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756012 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-data\") pod \"f2042f53-156d-42c2-9554-5fe559fbe56f\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " Oct 30 20:01:52.756204 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756123 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-envoy\") pod \"f2042f53-156d-42c2-9554-5fe559fbe56f\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " Oct 30 20:01:52.756204 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756163 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2042f53-156d-42c2-9554-5fe559fbe56f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2042f53-156d-42c2-9554-5fe559fbe56f-istiod-ca-cert\") pod \"f2042f53-156d-42c2-9554-5fe559fbe56f\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " Oct 30 20:01:52.756361 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756239 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-podinfo\") pod \"f2042f53-156d-42c2-9554-5fe559fbe56f\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " Oct 30 20:01:52.756361 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756245 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-workload-socket" pod "f2042f53-156d-42c2-9554-5fe559fbe56f" (UID: "f2042f53-156d-42c2-9554-5fe559fbe56f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:52.756361 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756281 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-workload-certs\") pod \"f2042f53-156d-42c2-9554-5fe559fbe56f\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " Oct 30 20:01:52.756361 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756314 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-data" pod "f2042f53-156d-42c2-9554-5fe559fbe56f" (UID: "f2042f53-156d-42c2-9554-5fe559fbe56f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:52.756555 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756364 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-credential-socket\") pod \"f2042f53-156d-42c2-9554-5fe559fbe56f\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " Oct 30 20:01:52.756555 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756421 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2042f53-156d-42c2-9554-5fe559fbe56f-kube-api-access-jh8zv\" (UniqueName: \"kubernetes.io/projected/f2042f53-156d-42c2-9554-5fe559fbe56f-kube-api-access-jh8zv\") pod \"f2042f53-156d-42c2-9554-5fe559fbe56f\" (UID: \"f2042f53-156d-42c2-9554-5fe559fbe56f\") " Oct 30 20:01:52.756555 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756482 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-workload-certs" pod "f2042f53-156d-42c2-9554-5fe559fbe56f" (UID: "f2042f53-156d-42c2-9554-5fe559fbe56f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:52.756838 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756598 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-credential-socket" pod "f2042f53-156d-42c2-9554-5fe559fbe56f" (UID: "f2042f53-156d-42c2-9554-5fe559fbe56f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:52.756838 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756639 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2042f53-156d-42c2-9554-5fe559fbe56f-istiod-ca-cert" pod "f2042f53-156d-42c2-9554-5fe559fbe56f" (UID: "f2042f53-156d-42c2-9554-5fe559fbe56f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:52.756943 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756916 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2042f53-156d-42c2-9554-5fe559fbe56f-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:52.756943 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756938 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:52.757097 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756952 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:52.757097 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756965 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:52.757097 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.756993 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:52.758344 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.758317 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-envoy" pod "f2042f53-156d-42c2-9554-5fe559fbe56f" (UID: "f2042f53-156d-42c2-9554-5fe559fbe56f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:52.758537 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.758518 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2042f53-156d-42c2-9554-5fe559fbe56f-kube-api-access-jh8zv" pod "f2042f53-156d-42c2-9554-5fe559fbe56f" (UID: "f2042f53-156d-42c2-9554-5fe559fbe56f"). InnerVolumeSpecName "kube-api-access-jh8zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:52.758730 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.758709 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-podinfo" pod "f2042f53-156d-42c2-9554-5fe559fbe56f" (UID: "f2042f53-156d-42c2-9554-5fe559fbe56f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:01:52.758827 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.758731 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-token" pod "f2042f53-156d-42c2-9554-5fe559fbe56f" (UID: "f2042f53-156d-42c2-9554-5fe559fbe56f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:52.858185 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.858143 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:52.858185 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.858184 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh8zv\" (UniqueName: \"kubernetes.io/projected/f2042f53-156d-42c2-9554-5fe559fbe56f-kube-api-access-jh8zv\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:52.858430 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.858200 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:52.858430 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:52.858214 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2042f53-156d-42c2-9554-5fe559fbe56f-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:53.122382 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:53.122343 4433 generic.go:356] "Generic (PLEG): container finished" podID="f2042f53-156d-42c2-9554-5fe559fbe56f" containerID="a7f3bcbd9ebc63079f1e74772b837350830b7c74bad2545cc97dbdaf8e120083" exitCode=0 Oct 30 20:01:53.122592 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:53.122408 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" Oct 30 20:01:53.122592 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:53.122437 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" event={"ID":"f2042f53-156d-42c2-9554-5fe559fbe56f","Type":"ContainerDied","Data":"a7f3bcbd9ebc63079f1e74772b837350830b7c74bad2545cc97dbdaf8e120083"} Oct 30 20:01:53.122592 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:53.122529 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx" event={"ID":"f2042f53-156d-42c2-9554-5fe559fbe56f","Type":"ContainerDied","Data":"469ff7aa6eade33ca277ac6ad8b5045aaf135b803b2dae2714d13f08e748a10c"} Oct 30 20:01:53.122592 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:53.122559 4433 scope.go:122] "RemoveContainer" containerID="a7f3bcbd9ebc63079f1e74772b837350830b7c74bad2545cc97dbdaf8e120083" Oct 30 20:01:53.132633 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:53.132612 4433 scope.go:122] "RemoveContainer" containerID="55e930b6f64912a78e94c389d42e9e1e413cd3c4b447eea781f0d496c6016fd3" Oct 30 20:01:53.141043 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:53.141026 4433 scope.go:122] "RemoveContainer" containerID="a7f3bcbd9ebc63079f1e74772b837350830b7c74bad2545cc97dbdaf8e120083" Oct 30 20:01:53.141297 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:01:53.141276 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f3bcbd9ebc63079f1e74772b837350830b7c74bad2545cc97dbdaf8e120083\": container with ID starting with a7f3bcbd9ebc63079f1e74772b837350830b7c74bad2545cc97dbdaf8e120083 not found: ID does not exist" containerID="a7f3bcbd9ebc63079f1e74772b837350830b7c74bad2545cc97dbdaf8e120083" Oct 30 20:01:53.141355 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:53.141308 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f3bcbd9ebc63079f1e74772b837350830b7c74bad2545cc97dbdaf8e120083"} err="failed to get container status \"a7f3bcbd9ebc63079f1e74772b837350830b7c74bad2545cc97dbdaf8e120083\": rpc error: code = NotFound desc = could not find container \"a7f3bcbd9ebc63079f1e74772b837350830b7c74bad2545cc97dbdaf8e120083\": container with ID starting with a7f3bcbd9ebc63079f1e74772b837350830b7c74bad2545cc97dbdaf8e120083 not found: ID does not exist" Oct 30 20:01:53.141355 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:53.141328 4433 scope.go:122] "RemoveContainer" containerID="55e930b6f64912a78e94c389d42e9e1e413cd3c4b447eea781f0d496c6016fd3" Oct 30 20:01:53.141587 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:01:53.141573 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e930b6f64912a78e94c389d42e9e1e413cd3c4b447eea781f0d496c6016fd3\": container with ID starting with 55e930b6f64912a78e94c389d42e9e1e413cd3c4b447eea781f0d496c6016fd3 not found: ID does not exist" containerID="55e930b6f64912a78e94c389d42e9e1e413cd3c4b447eea781f0d496c6016fd3" Oct 30 20:01:53.141643 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:53.141588 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e930b6f64912a78e94c389d42e9e1e413cd3c4b447eea781f0d496c6016fd3"} err="failed to get container status \"55e930b6f64912a78e94c389d42e9e1e413cd3c4b447eea781f0d496c6016fd3\": rpc error: code = NotFound desc = could not find container \"55e930b6f64912a78e94c389d42e9e1e413cd3c4b447eea781f0d496c6016fd3\": container with ID starting with 55e930b6f64912a78e94c389d42e9e1e413cd3c4b447eea781f0d496c6016fd3 not found: ID does not exist" Oct 30 20:01:53.286682 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:53.286645 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx"] Oct 30 20:01:53.289968 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:53.289935 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-r4nhx"] Oct 30 20:01:53.366617 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:01:53.366581 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:01:53.439777 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:53.439699 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:55.002320 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:55.002286 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:55.213689 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:55.213658 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2042f53-156d-42c2-9554-5fe559fbe56f" path="/var/lib/kubelet/pods/f2042f53-156d-42c2-9554-5fe559fbe56f/volumes" Oct 30 20:01:56.284828 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:56.284792 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:57.216171 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:57.216139 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:57.518439 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:57.518358 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:58.468112 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:58.468072 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb"] Oct 30 20:01:58.468331 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:58.468305 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" podUID="17419a84-960d-4236-90c3-debde62918d9" containerName="maas-api" containerID="cri-o://68a227d87457f866d9f4fc555d6b64f9f4213ad09ef0bc9795e63cc9ccabaac5" gracePeriod=30 Oct 30 20:01:58.664450 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:58.664416 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:58.731202 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:58.731149 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 20:01:58.836632 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:58.836493 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-metrics-certs\") pod \"17419a84-960d-4236-90c3-debde62918d9\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " Oct 30 20:01:58.842415 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:58.838313 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17419a84-960d-4236-90c3-debde62918d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17419a84-960d-4236-90c3-debde62918d9-openshift-service-ca\") pod \"17419a84-960d-4236-90c3-debde62918d9\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " Oct 30 20:01:58.842415 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:58.838422 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17419a84-960d-4236-90c3-debde62918d9-kube-api-access-wlvpc\" (UniqueName: \"kubernetes.io/projected/17419a84-960d-4236-90c3-debde62918d9-kube-api-access-wlvpc\") pod \"17419a84-960d-4236-90c3-debde62918d9\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " Oct 30 20:01:58.842415 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:58.838645 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-maas-api-tls\") pod \"17419a84-960d-4236-90c3-debde62918d9\" (UID: \"17419a84-960d-4236-90c3-debde62918d9\") " Oct 30 20:01:58.842415 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:58.840622 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17419a84-960d-4236-90c3-debde62918d9-openshift-service-ca" pod "17419a84-960d-4236-90c3-debde62918d9" (UID: "17419a84-960d-4236-90c3-debde62918d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:58.853714 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:58.843763 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17419a84-960d-4236-90c3-debde62918d9-kube-api-access-wlvpc" pod "17419a84-960d-4236-90c3-debde62918d9" (UID: "17419a84-960d-4236-90c3-debde62918d9"). InnerVolumeSpecName "kube-api-access-wlvpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:58.864757 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:58.862070 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-metrics-certs" pod "17419a84-960d-4236-90c3-debde62918d9" (UID: "17419a84-960d-4236-90c3-debde62918d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:58.864757 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:58.862497 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-maas-api-tls" pod "17419a84-960d-4236-90c3-debde62918d9" (UID: "17419a84-960d-4236-90c3-debde62918d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:58.939761 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:58.939729 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-maas-api-tls\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:58.939761 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:58.939760 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17419a84-960d-4236-90c3-debde62918d9-metrics-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:58.939761 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:58.939771 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17419a84-960d-4236-90c3-debde62918d9-openshift-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:58.939998 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:58.939779 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlvpc\" (UniqueName: \"kubernetes.io/projected/17419a84-960d-4236-90c3-debde62918d9-kube-api-access-wlvpc\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:01:59.021096 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:59.020967 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:59.156436 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:59.156402 4433 generic.go:356] "Generic (PLEG): container finished" podID="17419a84-960d-4236-90c3-debde62918d9" containerID="68a227d87457f866d9f4fc555d6b64f9f4213ad09ef0bc9795e63cc9ccabaac5" exitCode=0 Oct 30 20:01:59.156436 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:59.156446 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" event={"ID":"17419a84-960d-4236-90c3-debde62918d9","Type":"ContainerDied","Data":"68a227d87457f866d9f4fc555d6b64f9f4213ad09ef0bc9795e63cc9ccabaac5"} Oct 30 20:01:59.156651 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:59.156454 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" Oct 30 20:01:59.156651 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:59.156496 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb" event={"ID":"17419a84-960d-4236-90c3-debde62918d9","Type":"ContainerDied","Data":"2d08766e76c06fda19b5b4defae2fa3e2aa17206a25bfecd615363b2d3be871c"} Oct 30 20:01:59.156651 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:59.156527 4433 scope.go:122] "RemoveContainer" containerID="68a227d87457f866d9f4fc555d6b64f9f4213ad09ef0bc9795e63cc9ccabaac5" Oct 30 20:01:59.167875 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:59.167855 4433 scope.go:122] "RemoveContainer" containerID="68a227d87457f866d9f4fc555d6b64f9f4213ad09ef0bc9795e63cc9ccabaac5" Oct 30 20:01:59.168197 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:01:59.168176 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a227d87457f866d9f4fc555d6b64f9f4213ad09ef0bc9795e63cc9ccabaac5\": container with ID starting with 68a227d87457f866d9f4fc555d6b64f9f4213ad09ef0bc9795e63cc9ccabaac5 not found: ID does not exist" containerID="68a227d87457f866d9f4fc555d6b64f9f4213ad09ef0bc9795e63cc9ccabaac5" Oct 30 20:01:59.168274 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:59.168206 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a227d87457f866d9f4fc555d6b64f9f4213ad09ef0bc9795e63cc9ccabaac5"} err="failed to get container status \"68a227d87457f866d9f4fc555d6b64f9f4213ad09ef0bc9795e63cc9ccabaac5\": rpc error: code = NotFound desc = could not find container \"68a227d87457f866d9f4fc555d6b64f9f4213ad09ef0bc9795e63cc9ccabaac5\": container with ID starting with 68a227d87457f866d9f4fc555d6b64f9f4213ad09ef0bc9795e63cc9ccabaac5 not found: ID does not exist" Oct 30 20:01:59.334732 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:59.334699 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb"] Oct 30 20:01:59.339436 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:59.339389 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:59.340657 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:59.340632 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c75ed4-99f564d95-qg2kb"] Oct 30 20:01:59.963122 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:01:59.963086 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:00.241559 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:00.241478 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:00.552473 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:00.552382 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:00.790436 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:00.790405 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:01.163603 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:01.163556 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:01.212992 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:01.212947 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17419a84-960d-4236-90c3-debde62918d9" path="/var/lib/kubelet/pods/17419a84-960d-4236-90c3-debde62918d9/volumes" Oct 30 20:02:02.082480 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:02.082446 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7"] Oct 30 20:02:02.347621 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:02.347537 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:05.586969 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:05.586932 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:06.215323 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:06.215290 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:06.463523 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:06.463490 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:06.763579 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:06.763547 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:08.199971 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:08.199932 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:08.416518 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:08.416477 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth"] Oct 30 20:02:08.416764 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:08.416738 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" podUID="efa7df32-302f-44e6-aef2-c4aff86c2b78" containerName="istio-proxy" containerID="cri-o://bf43165051170dc959034f442f33f28976f4de0c826b1d0e5e998c6960a6ecbc" gracePeriod=30 Oct 30 20:02:08.507736 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:08.507636 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:08.789973 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:08.789890 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:09.068866 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:09.068834 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:11.040720 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:11.040685 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:12.286814 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:12.286774 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:12.537539 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:12.537452 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:13.480431 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.480386 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:13.686353 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.686285 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 20:02:13.709527 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.709485 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-data\") pod \"efa7df32-302f-44e6-aef2-c4aff86c2b78\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " Oct 30 20:02:13.709527 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.709520 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-workload-certs\") pod \"efa7df32-302f-44e6-aef2-c4aff86c2b78\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " Oct 30 20:02:13.709770 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.709558 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa7df32-302f-44e6-aef2-c4aff86c2b78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efa7df32-302f-44e6-aef2-c4aff86c2b78-istiod-ca-cert\") pod \"efa7df32-302f-44e6-aef2-c4aff86c2b78\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " Oct 30 20:02:13.709770 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.709577 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-workload-socket\") pod \"efa7df32-302f-44e6-aef2-c4aff86c2b78\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " Oct 30 20:02:13.709770 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.709615 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-envoy\") pod \"efa7df32-302f-44e6-aef2-c4aff86c2b78\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " Oct 30 20:02:13.709770 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.709655 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-podinfo\") pod \"efa7df32-302f-44e6-aef2-c4aff86c2b78\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " Oct 30 20:02:13.709770 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.709725 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa7df32-302f-44e6-aef2-c4aff86c2b78-kube-api-access-4gn8z\" (UniqueName: \"kubernetes.io/projected/efa7df32-302f-44e6-aef2-c4aff86c2b78-kube-api-access-4gn8z\") pod \"efa7df32-302f-44e6-aef2-c4aff86c2b78\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " Oct 30 20:02:13.710087 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.709835 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-data" pod "efa7df32-302f-44e6-aef2-c4aff86c2b78" (UID: "efa7df32-302f-44e6-aef2-c4aff86c2b78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:13.710087 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.709849 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-workload-certs" pod "efa7df32-302f-44e6-aef2-c4aff86c2b78" (UID: "efa7df32-302f-44e6-aef2-c4aff86c2b78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:13.710087 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.709864 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-credential-socket\") pod \"efa7df32-302f-44e6-aef2-c4aff86c2b78\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " Oct 30 20:02:13.710087 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.709947 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-token\" (UniqueName: \"kubernetes.io/projected/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-token\") pod \"efa7df32-302f-44e6-aef2-c4aff86c2b78\" (UID: \"efa7df32-302f-44e6-aef2-c4aff86c2b78\") " Oct 30 20:02:13.710087 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.710081 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-credential-socket" pod "efa7df32-302f-44e6-aef2-c4aff86c2b78" (UID: "efa7df32-302f-44e6-aef2-c4aff86c2b78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:13.710362 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.710109 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa7df32-302f-44e6-aef2-c4aff86c2b78-istiod-ca-cert" pod "efa7df32-302f-44e6-aef2-c4aff86c2b78" (UID: "efa7df32-302f-44e6-aef2-c4aff86c2b78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:13.710459 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.710433 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:13.710521 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.710468 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:13.710521 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.710483 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:13.710521 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.710496 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efa7df32-302f-44e6-aef2-c4aff86c2b78-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:13.710682 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.710564 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-workload-socket" pod "efa7df32-302f-44e6-aef2-c4aff86c2b78" (UID: "efa7df32-302f-44e6-aef2-c4aff86c2b78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:13.712517 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.712469 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa7df32-302f-44e6-aef2-c4aff86c2b78-kube-api-access-4gn8z" pod "efa7df32-302f-44e6-aef2-c4aff86c2b78" (UID: "efa7df32-302f-44e6-aef2-c4aff86c2b78"). InnerVolumeSpecName "kube-api-access-4gn8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:13.712663 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.712555 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-podinfo" pod "efa7df32-302f-44e6-aef2-c4aff86c2b78" (UID: "efa7df32-302f-44e6-aef2-c4aff86c2b78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:13.712663 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.712562 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-token" pod "efa7df32-302f-44e6-aef2-c4aff86c2b78" (UID: "efa7df32-302f-44e6-aef2-c4aff86c2b78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:13.712663 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.712644 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-envoy" pod "efa7df32-302f-44e6-aef2-c4aff86c2b78" (UID: "efa7df32-302f-44e6-aef2-c4aff86c2b78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:13.811693 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.811652 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:13.811693 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.811685 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:13.811693 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.811694 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:13.811935 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.811706 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gn8z\" (UniqueName: \"kubernetes.io/projected/efa7df32-302f-44e6-aef2-c4aff86c2b78-kube-api-access-4gn8z\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:13.811935 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.811716 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efa7df32-302f-44e6-aef2-c4aff86c2b78-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:13.849885 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.849849 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9"] Oct 30 20:02:13.899829 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:13.899791 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4"] Oct 30 20:02:14.024764 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:14.024676 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:14.246649 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:14.246616 4433 generic.go:356] "Generic (PLEG): container finished" podID="efa7df32-302f-44e6-aef2-c4aff86c2b78" containerID="bf43165051170dc959034f442f33f28976f4de0c826b1d0e5e998c6960a6ecbc" exitCode=0 Oct 30 20:02:14.246836 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:14.246667 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" Oct 30 20:02:14.246836 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:14.246693 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" event={"ID":"efa7df32-302f-44e6-aef2-c4aff86c2b78","Type":"ContainerDied","Data":"bf43165051170dc959034f442f33f28976f4de0c826b1d0e5e998c6960a6ecbc"} Oct 30 20:02:14.246836 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:14.246746 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" event={"ID":"efa7df32-302f-44e6-aef2-c4aff86c2b78","Type":"ContainerDied","Data":"1421e57bda8ca83dc57b31c4eb3664a5cf63ee795d9be6b4aa79ffba06151560"} Oct 30 20:02:14.246836 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:14.246772 4433 scope.go:122] "RemoveContainer" containerID="bf43165051170dc959034f442f33f28976f4de0c826b1d0e5e998c6960a6ecbc" Oct 30 20:02:14.256747 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:14.256729 4433 scope.go:122] "RemoveContainer" containerID="2841278e2d0b1dc47a4ca7dcea4fb66c1f7248b8418702608e13c991bd27ac41" Oct 30 20:02:14.265437 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:14.265416 4433 scope.go:122] "RemoveContainer" containerID="bf43165051170dc959034f442f33f28976f4de0c826b1d0e5e998c6960a6ecbc" Oct 30 20:02:14.265750 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:02:14.265731 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf43165051170dc959034f442f33f28976f4de0c826b1d0e5e998c6960a6ecbc\": container with ID starting with bf43165051170dc959034f442f33f28976f4de0c826b1d0e5e998c6960a6ecbc not found: ID does not exist" containerID="bf43165051170dc959034f442f33f28976f4de0c826b1d0e5e998c6960a6ecbc" Oct 30 20:02:14.265815 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:14.265761 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf43165051170dc959034f442f33f28976f4de0c826b1d0e5e998c6960a6ecbc"} err="failed to get container status \"bf43165051170dc959034f442f33f28976f4de0c826b1d0e5e998c6960a6ecbc\": rpc error: code = NotFound desc = could not find container \"bf43165051170dc959034f442f33f28976f4de0c826b1d0e5e998c6960a6ecbc\": container with ID starting with bf43165051170dc959034f442f33f28976f4de0c826b1d0e5e998c6960a6ecbc not found: ID does not exist" Oct 30 20:02:14.265815 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:14.265782 4433 scope.go:122] "RemoveContainer" containerID="2841278e2d0b1dc47a4ca7dcea4fb66c1f7248b8418702608e13c991bd27ac41" Oct 30 20:02:14.266006 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:02:14.265974 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2841278e2d0b1dc47a4ca7dcea4fb66c1f7248b8418702608e13c991bd27ac41\": container with ID starting with 2841278e2d0b1dc47a4ca7dcea4fb66c1f7248b8418702608e13c991bd27ac41 not found: ID does not exist" containerID="2841278e2d0b1dc47a4ca7dcea4fb66c1f7248b8418702608e13c991bd27ac41" Oct 30 20:02:14.266056 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:14.266010 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2841278e2d0b1dc47a4ca7dcea4fb66c1f7248b8418702608e13c991bd27ac41"} err="failed to get container status \"2841278e2d0b1dc47a4ca7dcea4fb66c1f7248b8418702608e13c991bd27ac41\": rpc error: code = NotFound desc = could not find container \"2841278e2d0b1dc47a4ca7dcea4fb66c1f7248b8418702608e13c991bd27ac41\": container with ID starting with 2841278e2d0b1dc47a4ca7dcea4fb66c1f7248b8418702608e13c991bd27ac41 not found: ID does not exist" Oct 30 20:02:14.410348 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:14.410310 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth"] Oct 30 20:02:14.415455 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:14.415418 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth"] Oct 30 20:02:14.625846 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:14.625799 4433 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:02:14.626278 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:14.625874 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-c75ed4-openshift-default-767c5c4d87-gwjth" podUID="efa7df32-302f-44e6-aef2-c4aff86c2b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:02:15.213469 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:15.213428 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efa7df32-302f-44e6-aef2-c4aff86c2b78" path="/var/lib/kubelet/pods/efa7df32-302f-44e6-aef2-c4aff86c2b78/volumes" Oct 30 20:02:16.078998 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:16.078953 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:17.102448 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.102394 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" podUID="7f1295b7-8682-4eed-b904-83b8b6537e8b" containerName="main" containerID="cri-o://90fc945efa9e5b9398d37a47ac70c5348a2bc0b570869f6a5b3fbf618fc6f38b" gracePeriod=45 Oct 30 20:02:17.264662 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.264627 4433 generic.go:356] "Generic (PLEG): container finished" podID="7f1295b7-8682-4eed-b904-83b8b6537e8b" containerID="90fc945efa9e5b9398d37a47ac70c5348a2bc0b570869f6a5b3fbf618fc6f38b" exitCode=0 Oct 30 20:02:17.264810 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.264703 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" event={"ID":"7f1295b7-8682-4eed-b904-83b8b6537e8b","Type":"ContainerDied","Data":"90fc945efa9e5b9398d37a47ac70c5348a2bc0b570869f6a5b3fbf618fc6f38b"} Oct 30 20:02:17.369025 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.368935 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 20:02:17.449769 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.449735 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f1295b7-8682-4eed-b904-83b8b6537e8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f1295b7-8682-4eed-b904-83b8b6537e8b-tls-certs\") pod \"7f1295b7-8682-4eed-b904-83b8b6537e8b\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " Oct 30 20:02:17.449973 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.449841 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-dshm\") pod \"7f1295b7-8682-4eed-b904-83b8b6537e8b\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " Oct 30 20:02:17.449973 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.449871 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-home\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-home\") pod \"7f1295b7-8682-4eed-b904-83b8b6537e8b\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " Oct 30 20:02:17.449973 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.449932 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f1295b7-8682-4eed-b904-83b8b6537e8b-kube-api-access-v8dzq\" (UniqueName: \"kubernetes.io/projected/7f1295b7-8682-4eed-b904-83b8b6537e8b-kube-api-access-v8dzq\") pod \"7f1295b7-8682-4eed-b904-83b8b6537e8b\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " Oct 30 20:02:17.449973 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.449950 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-tmp-dir\") pod \"7f1295b7-8682-4eed-b904-83b8b6537e8b\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " Oct 30 20:02:17.450247 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.450031 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-model-cache\") pod \"7f1295b7-8682-4eed-b904-83b8b6537e8b\" (UID: \"7f1295b7-8682-4eed-b904-83b8b6537e8b\") " Oct 30 20:02:17.450303 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.450223 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-home" pod "7f1295b7-8682-4eed-b904-83b8b6537e8b" (UID: "7f1295b7-8682-4eed-b904-83b8b6537e8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:17.450351 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.450309 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-tmp-dir" pod "7f1295b7-8682-4eed-b904-83b8b6537e8b" (UID: "7f1295b7-8682-4eed-b904-83b8b6537e8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:17.450397 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.450346 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-model-cache" pod "7f1295b7-8682-4eed-b904-83b8b6537e8b" (UID: "7f1295b7-8682-4eed-b904-83b8b6537e8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:17.450499 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.450480 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-tmp-dir\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:17.450533 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.450507 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-model-cache\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:17.450533 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.450523 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-home\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:17.452259 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.452227 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-dshm" pod "7f1295b7-8682-4eed-b904-83b8b6537e8b" (UID: "7f1295b7-8682-4eed-b904-83b8b6537e8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:17.452539 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.452515 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f1295b7-8682-4eed-b904-83b8b6537e8b-tls-certs" pod "7f1295b7-8682-4eed-b904-83b8b6537e8b" (UID: "7f1295b7-8682-4eed-b904-83b8b6537e8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:17.452642 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.452563 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1295b7-8682-4eed-b904-83b8b6537e8b-kube-api-access-v8dzq" pod "7f1295b7-8682-4eed-b904-83b8b6537e8b" (UID: "7f1295b7-8682-4eed-b904-83b8b6537e8b"). InnerVolumeSpecName "kube-api-access-v8dzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:17.551330 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.551293 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8dzq\" (UniqueName: \"kubernetes.io/projected/7f1295b7-8682-4eed-b904-83b8b6537e8b-kube-api-access-v8dzq\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:17.551330 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.551325 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f1295b7-8682-4eed-b904-83b8b6537e8b-tls-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:17.551330 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:17.551334 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f1295b7-8682-4eed-b904-83b8b6537e8b-dshm\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:18.271711 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:18.271679 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" Oct 30 20:02:18.272212 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:18.271715 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7" event={"ID":"7f1295b7-8682-4eed-b904-83b8b6537e8b","Type":"ContainerDied","Data":"e88ecb19b9aff58f14ecda10607c60f3873edf48e1c001c144ee5ef5ffa1e114"} Oct 30 20:02:18.272212 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:18.271770 4433 scope.go:122] "RemoveContainer" containerID="90fc945efa9e5b9398d37a47ac70c5348a2bc0b570869f6a5b3fbf618fc6f38b" Oct 30 20:02:18.436747 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:18.436708 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7"] Oct 30 20:02:18.485642 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:18.485608 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/distinct-2-w2-a3f9b7-kserve-546855b4b-qrhz7"] Oct 30 20:02:18.487801 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:18.487768 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7"] Oct 30 20:02:19.009403 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:19.009364 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:19.213551 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:19.213507 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f1295b7-8682-4eed-b904-83b8b6537e8b" path="/var/lib/kubelet/pods/7f1295b7-8682-4eed-b904-83b8b6537e8b/volumes" Oct 30 20:02:19.263591 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:19.263498 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:20.782560 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.782521 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb"] Oct 30 20:02:20.783219 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783198 4433 cpu_manager.go:414] "containerMap: removing container" podUID="3af9891c-2337-4531-a662-1eb43724b440" containerName="istio-proxy" Oct 30 20:02:20.783318 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783220 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3af9891c-2337-4531-a662-1eb43724b440" containerName="istio-proxy" Oct 30 20:02:20.783318 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783239 4433 cpu_manager.go:414] "containerMap: removing container" podUID="eafb6a50-b7db-47d6-b742-86b92bfef67d" containerName="main" Oct 30 20:02:20.783318 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783246 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafb6a50-b7db-47d6-b742-86b92bfef67d" containerName="main" Oct 30 20:02:20.783318 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783261 4433 cpu_manager.go:414] "containerMap: removing container" podUID="34f8f88e-3537-4e62-ae87-b12ec3812f62" containerName="payload-processing" Oct 30 20:02:20.783318 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783269 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f8f88e-3537-4e62-ae87-b12ec3812f62" containerName="payload-processing" Oct 30 20:02:20.783318 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783287 4433 cpu_manager.go:414] "containerMap: removing container" podUID="e60baa7a-9536-4dd5-b59c-d042cd41b798" containerName="payload-pre-processing" Oct 30 20:02:20.783318 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783295 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e60baa7a-9536-4dd5-b59c-d042cd41b798" containerName="payload-pre-processing" Oct 30 20:02:20.783318 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783305 4433 cpu_manager.go:414] "containerMap: removing container" podUID="2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" containerName="cleanup" Oct 30 20:02:20.783318 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783313 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783323 4433 cpu_manager.go:414] "containerMap: removing container" podUID="ff913782-d7e9-4110-a499-623fda834acd" containerName="istio-proxy" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783332 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff913782-d7e9-4110-a499-623fda834acd" containerName="istio-proxy" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783345 4433 cpu_manager.go:414] "containerMap: removing container" podUID="f2042f53-156d-42c2-9554-5fe559fbe56f" containerName="istio-proxy" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783352 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2042f53-156d-42c2-9554-5fe559fbe56f" containerName="istio-proxy" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783362 4433 cpu_manager.go:414] "containerMap: removing container" podUID="ded326eb-7db6-432f-8427-68833723a5fa" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783369 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded326eb-7db6-432f-8427-68833723a5fa" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783379 4433 cpu_manager.go:414] "containerMap: removing container" podUID="ded326eb-7db6-432f-8427-68833723a5fa" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783385 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded326eb-7db6-432f-8427-68833723a5fa" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783399 4433 cpu_manager.go:414] "containerMap: removing container" podUID="7f1295b7-8682-4eed-b904-83b8b6537e8b" containerName="main" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783408 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f1295b7-8682-4eed-b904-83b8b6537e8b" containerName="main" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783419 4433 cpu_manager.go:414] "containerMap: removing container" podUID="17419a84-960d-4236-90c3-debde62918d9" containerName="maas-api" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783426 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17419a84-960d-4236-90c3-debde62918d9" containerName="maas-api" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783435 4433 cpu_manager.go:414] "containerMap: removing container" podUID="efa7df32-302f-44e6-aef2-c4aff86c2b78" containerName="istio-proxy" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783442 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa7df32-302f-44e6-aef2-c4aff86c2b78" containerName="istio-proxy" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783455 4433 cpu_manager.go:414] "containerMap: removing container" podUID="56448e26-174f-4edd-96a7-87cbd714ad8e" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783462 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56448e26-174f-4edd-96a7-87cbd714ad8e" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783476 4433 cpu_manager.go:414] "containerMap: removing container" podUID="aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0" containerName="maas-api" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783484 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0" containerName="maas-api" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783496 4433 cpu_manager.go:414] "containerMap: removing container" podUID="56448e26-174f-4edd-96a7-87cbd714ad8e" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783502 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56448e26-174f-4edd-96a7-87cbd714ad8e" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783513 4433 cpu_manager.go:414] "containerMap: removing container" podUID="3af9891c-2337-4531-a662-1eb43724b440" containerName="istio-proxy" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783520 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3af9891c-2337-4531-a662-1eb43724b440" containerName="istio-proxy" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783531 4433 cpu_manager.go:414] "containerMap: removing container" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerName="main" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783537 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerName="main" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783548 4433 cpu_manager.go:414] "containerMap: removing container" podUID="56448e26-174f-4edd-96a7-87cbd714ad8e" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783555 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56448e26-174f-4edd-96a7-87cbd714ad8e" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783570 4433 cpu_manager.go:414] "containerMap: removing container" podUID="2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783577 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783593 4433 cpu_manager.go:414] "containerMap: removing container" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783601 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783614 4433 cpu_manager.go:414] "containerMap: removing container" podUID="f2042f53-156d-42c2-9554-5fe559fbe56f" containerName="istio-proxy" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783621 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2042f53-156d-42c2-9554-5fe559fbe56f" containerName="istio-proxy" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783632 4433 cpu_manager.go:414] "containerMap: removing container" podUID="f1e0904a-f62b-4828-b130-fdef755b3831" containerName="istio-proxy" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783640 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e0904a-f62b-4828-b130-fdef755b3831" containerName="istio-proxy" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783655 4433 cpu_manager.go:414] "containerMap: removing container" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783663 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783673 4433 cpu_manager.go:414] "containerMap: removing container" podUID="37be5208-015f-4ab4-8bd0-f390d3729293" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783680 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37be5208-015f-4ab4-8bd0-f390d3729293" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783695 4433 cpu_manager.go:414] "containerMap: removing container" podUID="37be5208-015f-4ab4-8bd0-f390d3729293" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783701 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37be5208-015f-4ab4-8bd0-f390d3729293" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783711 4433 cpu_manager.go:414] "containerMap: removing container" podUID="37be5208-015f-4ab4-8bd0-f390d3729293" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783718 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37be5208-015f-4ab4-8bd0-f390d3729293" containerName="cleanup" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783738 4433 cpu_manager.go:414] "containerMap: removing container" podUID="d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" containerName="istio-proxy" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783746 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" containerName="istio-proxy" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783755 4433 cpu_manager.go:414] "containerMap: removing container" podUID="1550c04a-cebb-456c-9a05-4eeddaefef8a" containerName="istio-proxy" Oct 30 20:02:20.783775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783762 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1550c04a-cebb-456c-9a05-4eeddaefef8a" containerName="istio-proxy" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783875 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f1295b7-8682-4eed-b904-83b8b6537e8b" containerName="main" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783888 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="17419a84-960d-4236-90c3-debde62918d9" containerName="maas-api" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783897 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa7df32-302f-44e6-aef2-c4aff86c2b78" containerName="istio-proxy" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783907 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="56448e26-174f-4edd-96a7-87cbd714ad8e" containerName="cleanup" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783916 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa49aae0-aa9c-43e4-a3cd-d27cb46a5fd0" containerName="maas-api" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783926 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="56448e26-174f-4edd-96a7-87cbd714ad8e" containerName="cleanup" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783934 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3af9891c-2337-4531-a662-1eb43724b440" containerName="istio-proxy" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783941 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9c64c5-157e-4750-9a42-3c2b7f15e741" containerName="main" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783951 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="56448e26-174f-4edd-96a7-87cbd714ad8e" containerName="cleanup" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783958 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" containerName="cleanup" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783969 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c6dccf0-37fd-41f6-ae17-ab1e44c13c89" containerName="main" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.783995 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa7df32-302f-44e6-aef2-c4aff86c2b78" containerName="istio-proxy" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784004 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff913782-d7e9-4110-a499-623fda834acd" containerName="istio-proxy" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784011 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2042f53-156d-42c2-9554-5fe559fbe56f" containerName="istio-proxy" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784020 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded326eb-7db6-432f-8427-68833723a5fa" containerName="cleanup" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784028 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e0904a-f62b-4828-b130-fdef755b3831" containerName="istio-proxy" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784037 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b9f088-a105-4af4-82ad-3efdf214a5e2" containerName="main" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784047 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="37be5208-015f-4ab4-8bd0-f390d3729293" containerName="cleanup" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784056 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e0904a-f62b-4828-b130-fdef755b3831" containerName="istio-proxy" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784064 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="37be5208-015f-4ab4-8bd0-f390d3729293" containerName="cleanup" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784071 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="37be5208-015f-4ab4-8bd0-f390d3729293" containerName="cleanup" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784083 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d00c90c8-e436-49e8-aee6-e5f4c7a38cb4" containerName="istio-proxy" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784091 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1550c04a-cebb-456c-9a05-4eeddaefef8a" containerName="istio-proxy" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784101 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafb6a50-b7db-47d6-b742-86b92bfef67d" containerName="main" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784111 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f8f88e-3537-4e62-ae87-b12ec3812f62" containerName="payload-processing" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784120 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e60baa7a-9536-4dd5-b59c-d042cd41b798" containerName="payload-pre-processing" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784131 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" containerName="cleanup" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784138 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2042f53-156d-42c2-9554-5fe559fbe56f" containerName="istio-proxy" Oct 30 20:02:20.785962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.784147 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded326eb-7db6-432f-8427-68833723a5fa" containerName="cleanup" Oct 30 20:02:20.789443 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.789420 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" Oct 30 20:02:20.845100 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.845067 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:02:20.847044 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.847012 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb"] Oct 30 20:02:20.890890 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.890854 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxdb\" (UniqueName: \"kubernetes.io/projected/d9390ac4-cd34-46fa-ae2d-491bd2668c99-kube-api-access-hzxdb\") pod \"maas-api-revoke-sub-ddb5f24e-66ghb\" (UID: \"d9390ac4-cd34-46fa-ae2d-491bd2668c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" Oct 30 20:02:20.891111 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.890915 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9390ac4-cd34-46fa-ae2d-491bd2668c99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddb5f24e-66ghb\" (UID: \"d9390ac4-cd34-46fa-ae2d-491bd2668c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" Oct 30 20:02:20.891201 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.891152 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9390ac4-cd34-46fa-ae2d-491bd2668c99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddb5f24e-66ghb\" (UID: \"d9390ac4-cd34-46fa-ae2d-491bd2668c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" Oct 30 20:02:20.893517 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.893499 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\"" Oct 30 20:02:20.992382 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.992346 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9390ac4-cd34-46fa-ae2d-491bd2668c99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddb5f24e-66ghb\" (UID: \"d9390ac4-cd34-46fa-ae2d-491bd2668c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" Oct 30 20:02:20.992557 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.992498 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9390ac4-cd34-46fa-ae2d-491bd2668c99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddb5f24e-66ghb\" (UID: \"d9390ac4-cd34-46fa-ae2d-491bd2668c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" Oct 30 20:02:20.992654 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.992630 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxdb\" (UniqueName: \"kubernetes.io/projected/d9390ac4-cd34-46fa-ae2d-491bd2668c99-kube-api-access-hzxdb\") pod \"maas-api-revoke-sub-ddb5f24e-66ghb\" (UID: \"d9390ac4-cd34-46fa-ae2d-491bd2668c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" Oct 30 20:02:20.993202 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.993181 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9390ac4-cd34-46fa-ae2d-491bd2668c99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddb5f24e-66ghb\" (UID: \"d9390ac4-cd34-46fa-ae2d-491bd2668c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" Oct 30 20:02:20.994786 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:20.994759 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9390ac4-cd34-46fa-ae2d-491bd2668c99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddb5f24e-66ghb\" (UID: \"d9390ac4-cd34-46fa-ae2d-491bd2668c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" Oct 30 20:02:21.049921 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:21.049849 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxdb\" (UniqueName: \"kubernetes.io/projected/d9390ac4-cd34-46fa-ae2d-491bd2668c99-kube-api-access-hzxdb\") pod \"maas-api-revoke-sub-ddb5f24e-66ghb\" (UID: \"d9390ac4-cd34-46fa-ae2d-491bd2668c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" Oct 30 20:02:21.104233 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:21.104198 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" Oct 30 20:02:21.135104 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:21.135069 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb"] Oct 30 20:02:21.242582 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:21.242546 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt"] Oct 30 20:02:21.481518 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:21.481487 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb"] Oct 30 20:02:21.483933 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:02:21.483899 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9390ac4_cd34_46fa_ae2d_491bd2668c99.slice/crio-2893a66c8be7a57ca7613a95796ae2888038b6039efa93d487da0f4fe9eabfa8 WatchSource:0}: Error finding container 2893a66c8be7a57ca7613a95796ae2888038b6039efa93d487da0f4fe9eabfa8: Status 404 returned error can't find the container with id 2893a66c8be7a57ca7613a95796ae2888038b6039efa93d487da0f4fe9eabfa8 Oct 30 20:02:21.601875 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:21.601842 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:21.829615 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:21.829572 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:22.299511 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:22.299472 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" event={"ID":"d9390ac4-cd34-46fa-ae2d-491bd2668c99","Type":"ContainerStarted","Data":"73b466979f0d630e737fef99653e68fe06425bcebd90e10d7be05fa84dfa2c48"} Oct 30 20:02:22.299698 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:22.299532 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" event={"ID":"d9390ac4-cd34-46fa-ae2d-491bd2668c99","Type":"ContainerStarted","Data":"2893a66c8be7a57ca7613a95796ae2888038b6039efa93d487da0f4fe9eabfa8"} Oct 30 20:02:22.419875 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:22.419818 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" podStartSLOduration=2.419804083 podStartE2EDuration="2.419804083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:22.417088251 +0000 UTC m=+2789.351875776" watchObservedRunningTime="2026-10-30 20:02:22.419804083 +0000 UTC m=+2789.354591607" Oct 30 20:02:23.305946 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:23.305912 4433 generic.go:356] "Generic (PLEG): container finished" podID="d9390ac4-cd34-46fa-ae2d-491bd2668c99" containerID="73b466979f0d630e737fef99653e68fe06425bcebd90e10d7be05fa84dfa2c48" exitCode=0 Oct 30 20:02:23.306368 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:23.306015 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" event={"ID":"d9390ac4-cd34-46fa-ae2d-491bd2668c99","Type":"ContainerDied","Data":"73b466979f0d630e737fef99653e68fe06425bcebd90e10d7be05fa84dfa2c48"} Oct 30 20:02:23.368026 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:02:23.367963 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:02:23.488811 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:23.488766 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" podUID="23196d8b-4348-4e50-9f54-5c44c4b6b35c" containerName="payload-processing" containerID="cri-o://76f3117e773731c930e8598876b7bf56cc0c59a035332a34ef139c5149ff525d" gracePeriod=25 Oct 30 20:02:24.453557 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:24.453513 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" podUID="23196d8b-4348-4e50-9f54-5c44c4b6b35c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:24.458727 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:24.458696 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" Oct 30 20:02:24.553530 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:24.553492 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9390ac4-cd34-46fa-ae2d-491bd2668c99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9390ac4-cd34-46fa-ae2d-491bd2668c99-maas-api-xks-ca\") pod \"d9390ac4-cd34-46fa-ae2d-491bd2668c99\" (UID: \"d9390ac4-cd34-46fa-ae2d-491bd2668c99\") " Oct 30 20:02:24.553703 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:24.553545 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9390ac4-cd34-46fa-ae2d-491bd2668c99-kube-api-access-hzxdb\" (UniqueName: \"kubernetes.io/projected/d9390ac4-cd34-46fa-ae2d-491bd2668c99-kube-api-access-hzxdb\") pod \"d9390ac4-cd34-46fa-ae2d-491bd2668c99\" (UID: \"d9390ac4-cd34-46fa-ae2d-491bd2668c99\") " Oct 30 20:02:24.553703 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:24.553622 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9390ac4-cd34-46fa-ae2d-491bd2668c99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9390ac4-cd34-46fa-ae2d-491bd2668c99-maas-api-openshift-ca\") pod \"d9390ac4-cd34-46fa-ae2d-491bd2668c99\" (UID: \"d9390ac4-cd34-46fa-ae2d-491bd2668c99\") " Oct 30 20:02:24.554060 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:24.554033 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9390ac4-cd34-46fa-ae2d-491bd2668c99-maas-api-openshift-ca" pod "d9390ac4-cd34-46fa-ae2d-491bd2668c99" (UID: "d9390ac4-cd34-46fa-ae2d-491bd2668c99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:24.555765 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:24.555735 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9390ac4-cd34-46fa-ae2d-491bd2668c99-kube-api-access-hzxdb" pod "d9390ac4-cd34-46fa-ae2d-491bd2668c99" (UID: "d9390ac4-cd34-46fa-ae2d-491bd2668c99"). InnerVolumeSpecName "kube-api-access-hzxdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:24.555871 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:24.555831 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9390ac4-cd34-46fa-ae2d-491bd2668c99-maas-api-xks-ca" pod "d9390ac4-cd34-46fa-ae2d-491bd2668c99" (UID: "d9390ac4-cd34-46fa-ae2d-491bd2668c99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:24.655236 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:24.655189 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9390ac4-cd34-46fa-ae2d-491bd2668c99-maas-api-xks-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:24.655236 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:24.655227 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzxdb\" (UniqueName: \"kubernetes.io/projected/d9390ac4-cd34-46fa-ae2d-491bd2668c99-kube-api-access-hzxdb\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:24.655236 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:24.655237 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9390ac4-cd34-46fa-ae2d-491bd2668c99-maas-api-openshift-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:25.319342 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:25.319307 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" event={"ID":"d9390ac4-cd34-46fa-ae2d-491bd2668c99","Type":"ContainerDied","Data":"2893a66c8be7a57ca7613a95796ae2888038b6039efa93d487da0f4fe9eabfa8"} Oct 30 20:02:25.319342 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:25.319339 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb" Oct 30 20:02:25.319567 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:25.319365 4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2893a66c8be7a57ca7613a95796ae2888038b6039efa93d487da0f4fe9eabfa8" Oct 30 20:02:26.136303 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:26.136236 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" podUID="c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904" containerName="payload-processing" containerID="cri-o://dec892ca2f8293c803cc61367f0c51afc33e8ecfb35f939fa37e9cf3cf684b3f" gracePeriod=25 Oct 30 20:02:26.243218 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:26.243164 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" podUID="edac81ed-b1b2-4ddb-a2ae-bfe897991191" containerName="payload-pre-processing" containerID="cri-o://1d8fe6454a3d87a0b050c54a7ce2119117ecebc4d202e0746e7322c8316c8fc2" gracePeriod=25 Oct 30 20:02:26.316289 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:26.316252 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:28.329994 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:28.329939 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" podUID="edac81ed-b1b2-4ddb-a2ae-bfe897991191" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:28.330418 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:28.329936 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" podUID="c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:28.870576 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:28.870530 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" podUID="1bd1c66b-4d08-4aca-ab6b-05b7267e7821" containerName="main" containerID="cri-o://95b8e7d5e3dc77cd6c1c00ac9d5d4cdb28d0f0195e44ce0c35163d326614cfa0" gracePeriod=45 Oct 30 20:02:28.923823 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:28.923777 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" podUID="c2ffbe2b-c076-4ae7-994b-d8ab2db02185" containerName="main" containerID="cri-o://0a28952560c20e4662b52d0e9ecd0bc4511843fb8313a36c88fcdc7dddc8fcb0" gracePeriod=45 Oct 30 20:02:29.200449 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.200414 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 20:02:29.222209 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.222178 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 20:02:29.323783 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.323745 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-kube-api-access-756rx\" (UniqueName: \"kubernetes.io/projected/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-kube-api-access-756rx\") pod \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " Oct 30 20:02:29.324003 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.323793 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-kube-api-access-57hqz\" (UniqueName: \"kubernetes.io/projected/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-kube-api-access-57hqz\") pod \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " Oct 30 20:02:29.324003 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.323841 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-tls-certs\") pod \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " Oct 30 20:02:29.324003 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.323878 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-tmp-dir\") pod \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " Oct 30 20:02:29.324003 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.323913 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-model-cache\") pod \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " Oct 30 20:02:29.324003 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.323952 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-dshm\") pod \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " Oct 30 20:02:29.324288 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.324036 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-home\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-home\") pod \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " Oct 30 20:02:29.324288 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.324165 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-home\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-home\") pod \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " Oct 30 20:02:29.324288 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.324206 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-dshm\") pod \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " Oct 30 20:02:29.324288 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.324240 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-model-cache\") pod \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\" (UID: \"c2ffbe2b-c076-4ae7-994b-d8ab2db02185\") " Oct 30 20:02:29.324288 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.324271 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-tmp-dir\") pod \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " Oct 30 20:02:29.324535 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.324327 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-tls-certs\") pod \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\" (UID: \"1bd1c66b-4d08-4aca-ab6b-05b7267e7821\") " Oct 30 20:02:29.324535 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.324232 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-model-cache" pod "1bd1c66b-4d08-4aca-ab6b-05b7267e7821" (UID: "1bd1c66b-4d08-4aca-ab6b-05b7267e7821"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:29.324642 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.324545 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-home" pod "c2ffbe2b-c076-4ae7-994b-d8ab2db02185" (UID: "c2ffbe2b-c076-4ae7-994b-d8ab2db02185"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:29.324853 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.324826 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-model-cache" pod "c2ffbe2b-c076-4ae7-994b-d8ab2db02185" (UID: "c2ffbe2b-c076-4ae7-994b-d8ab2db02185"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:29.325085 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.325059 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-tmp-dir" pod "1bd1c66b-4d08-4aca-ab6b-05b7267e7821" (UID: "1bd1c66b-4d08-4aca-ab6b-05b7267e7821"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:29.325222 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.324249 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-tmp-dir" pod "c2ffbe2b-c076-4ae7-994b-d8ab2db02185" (UID: "c2ffbe2b-c076-4ae7-994b-d8ab2db02185"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:29.325222 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.324440 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-home" pod "1bd1c66b-4d08-4aca-ab6b-05b7267e7821" (UID: "1bd1c66b-4d08-4aca-ab6b-05b7267e7821"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:29.325222 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.325167 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-model-cache\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:29.325392 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.325278 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-home\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:29.328587 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.328447 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-dshm" pod "c2ffbe2b-c076-4ae7-994b-d8ab2db02185" (UID: "c2ffbe2b-c076-4ae7-994b-d8ab2db02185"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:29.328587 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.328558 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-dshm" pod "1bd1c66b-4d08-4aca-ab6b-05b7267e7821" (UID: "1bd1c66b-4d08-4aca-ab6b-05b7267e7821"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:29.328784 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.328591 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-kube-api-access-57hqz" pod "1bd1c66b-4d08-4aca-ab6b-05b7267e7821" (UID: "1bd1c66b-4d08-4aca-ab6b-05b7267e7821"). InnerVolumeSpecName "kube-api-access-57hqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:29.328784 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.328627 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-tls-certs" pod "1bd1c66b-4d08-4aca-ab6b-05b7267e7821" (UID: "1bd1c66b-4d08-4aca-ab6b-05b7267e7821"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:29.328784 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.328642 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-tls-certs" pod "c2ffbe2b-c076-4ae7-994b-d8ab2db02185" (UID: "c2ffbe2b-c076-4ae7-994b-d8ab2db02185"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:29.329068 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.329043 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-kube-api-access-756rx" pod "c2ffbe2b-c076-4ae7-994b-d8ab2db02185" (UID: "c2ffbe2b-c076-4ae7-994b-d8ab2db02185"). InnerVolumeSpecName "kube-api-access-756rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:29.342594 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.342560 4433 generic.go:356] "Generic (PLEG): container finished" podID="1bd1c66b-4d08-4aca-ab6b-05b7267e7821" containerID="95b8e7d5e3dc77cd6c1c00ac9d5d4cdb28d0f0195e44ce0c35163d326614cfa0" exitCode=0 Oct 30 20:02:29.343104 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.342605 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" Oct 30 20:02:29.343104 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.342643 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" event={"ID":"1bd1c66b-4d08-4aca-ab6b-05b7267e7821","Type":"ContainerDied","Data":"95b8e7d5e3dc77cd6c1c00ac9d5d4cdb28d0f0195e44ce0c35163d326614cfa0"} Oct 30 20:02:29.343104 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.342685 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9" event={"ID":"1bd1c66b-4d08-4aca-ab6b-05b7267e7821","Type":"ContainerDied","Data":"3567244eb522388b4bebf1bab633caeb81a9bc770e84b37cbff8efa20bf07d0c"} Oct 30 20:02:29.343104 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.342712 4433 scope.go:122] "RemoveContainer" containerID="95b8e7d5e3dc77cd6c1c00ac9d5d4cdb28d0f0195e44ce0c35163d326614cfa0" Oct 30 20:02:29.344152 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.344129 4433 generic.go:356] "Generic (PLEG): container finished" podID="c2ffbe2b-c076-4ae7-994b-d8ab2db02185" containerID="0a28952560c20e4662b52d0e9ecd0bc4511843fb8313a36c88fcdc7dddc8fcb0" exitCode=0 Oct 30 20:02:29.344245 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.344178 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" Oct 30 20:02:29.344245 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.344206 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" event={"ID":"c2ffbe2b-c076-4ae7-994b-d8ab2db02185","Type":"ContainerDied","Data":"0a28952560c20e4662b52d0e9ecd0bc4511843fb8313a36c88fcdc7dddc8fcb0"} Oct 30 20:02:29.344330 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.344247 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4" event={"ID":"c2ffbe2b-c076-4ae7-994b-d8ab2db02185","Type":"ContainerDied","Data":"094f3b6c5219b1bba85b73ab8290313b41b854adcf6fd44a8ce0e51cdac1af94"} Oct 30 20:02:29.354450 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.354428 4433 scope.go:122] "RemoveContainer" containerID="95b8e7d5e3dc77cd6c1c00ac9d5d4cdb28d0f0195e44ce0c35163d326614cfa0" Oct 30 20:02:29.354818 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:02:29.354781 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b8e7d5e3dc77cd6c1c00ac9d5d4cdb28d0f0195e44ce0c35163d326614cfa0\": container with ID starting with 95b8e7d5e3dc77cd6c1c00ac9d5d4cdb28d0f0195e44ce0c35163d326614cfa0 not found: ID does not exist" containerID="95b8e7d5e3dc77cd6c1c00ac9d5d4cdb28d0f0195e44ce0c35163d326614cfa0" Oct 30 20:02:29.354879 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.354811 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b8e7d5e3dc77cd6c1c00ac9d5d4cdb28d0f0195e44ce0c35163d326614cfa0"} err="failed to get container status \"95b8e7d5e3dc77cd6c1c00ac9d5d4cdb28d0f0195e44ce0c35163d326614cfa0\": rpc error: code = NotFound desc = could not find container \"95b8e7d5e3dc77cd6c1c00ac9d5d4cdb28d0f0195e44ce0c35163d326614cfa0\": container with ID starting with 95b8e7d5e3dc77cd6c1c00ac9d5d4cdb28d0f0195e44ce0c35163d326614cfa0 not found: ID does not exist" Oct 30 20:02:29.354879 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.354838 4433 scope.go:122] "RemoveContainer" containerID="0a28952560c20e4662b52d0e9ecd0bc4511843fb8313a36c88fcdc7dddc8fcb0" Oct 30 20:02:29.364166 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.364142 4433 scope.go:122] "RemoveContainer" containerID="0a28952560c20e4662b52d0e9ecd0bc4511843fb8313a36c88fcdc7dddc8fcb0" Oct 30 20:02:29.364521 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:02:29.364500 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a28952560c20e4662b52d0e9ecd0bc4511843fb8313a36c88fcdc7dddc8fcb0\": container with ID starting with 0a28952560c20e4662b52d0e9ecd0bc4511843fb8313a36c88fcdc7dddc8fcb0 not found: ID does not exist" containerID="0a28952560c20e4662b52d0e9ecd0bc4511843fb8313a36c88fcdc7dddc8fcb0" Oct 30 20:02:29.364593 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.364529 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a28952560c20e4662b52d0e9ecd0bc4511843fb8313a36c88fcdc7dddc8fcb0"} err="failed to get container status \"0a28952560c20e4662b52d0e9ecd0bc4511843fb8313a36c88fcdc7dddc8fcb0\": rpc error: code = NotFound desc = could not find container \"0a28952560c20e4662b52d0e9ecd0bc4511843fb8313a36c88fcdc7dddc8fcb0\": container with ID starting with 0a28952560c20e4662b52d0e9ecd0bc4511843fb8313a36c88fcdc7dddc8fcb0 not found: ID does not exist" Oct 30 20:02:29.425910 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.425824 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-dshm\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:29.425910 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.425851 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-home\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:29.425910 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.425860 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-dshm\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:29.425910 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.425869 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-model-cache\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:29.425910 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.425879 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-tmp-dir\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:29.425910 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.425888 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-tls-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:29.425910 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.425899 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-756rx\" (UniqueName: \"kubernetes.io/projected/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-kube-api-access-756rx\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:29.425910 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.425907 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57hqz\" (UniqueName: \"kubernetes.io/projected/1bd1c66b-4d08-4aca-ab6b-05b7267e7821-kube-api-access-57hqz\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:29.425910 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.425917 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-tls-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:29.426373 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.425925 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ffbe2b-c076-4ae7-994b-d8ab2db02185-tmp-dir\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:29.540713 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.540633 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9"] Oct 30 20:02:29.543255 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.543222 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/distinct-2-w3-ada8d7-kserve-b47dd4f88-x48k9"] Oct 30 20:02:29.652891 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.652841 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4"] Oct 30 20:02:29.655207 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:29.655175 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/premium-w3-ada8d7-kserve-fcdf88bb4-nh9q4"] Oct 30 20:02:31.096875 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:31.096834 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:31.213776 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:31.213742 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bd1c66b-4d08-4aca-ab6b-05b7267e7821" path="/var/lib/kubelet/pods/1bd1c66b-4d08-4aca-ab6b-05b7267e7821/volumes" Oct 30 20:02:31.214233 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:31.214216 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2ffbe2b-c076-4ae7-994b-d8ab2db02185" path="/var/lib/kubelet/pods/c2ffbe2b-c076-4ae7-994b-d8ab2db02185/volumes" Oct 30 20:02:31.605574 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:31.605532 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:32.513423 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:32.513385 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:34.453354 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:34.453315 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" podUID="23196d8b-4348-4e50-9f54-5c44c4b6b35c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:35.743960 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:35.743921 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:38.329479 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:38.329431 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" podUID="edac81ed-b1b2-4ddb-a2ae-bfe897991191" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:38.330124 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:38.329518 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" podUID="c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:38.780807 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:38.780712 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:39.008760 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:39.008729 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:39.857658 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:39.857613 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:40.550730 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:40.550697 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:41.846268 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:41.846229 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:43.755404 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:43.755314 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" Oct 30 20:02:43.880353 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:43.880319 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23196d8b-4348-4e50-9f54-5c44c4b6b35c-kube-api-access-xr8td\" (UniqueName: \"kubernetes.io/projected/23196d8b-4348-4e50-9f54-5c44c4b6b35c-kube-api-access-xr8td\") pod \"23196d8b-4348-4e50-9f54-5c44c4b6b35c\" (UID: \"23196d8b-4348-4e50-9f54-5c44c4b6b35c\") " Oct 30 20:02:43.880533 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:43.880395 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23196d8b-4348-4e50-9f54-5c44c4b6b35c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23196d8b-4348-4e50-9f54-5c44c4b6b35c-plugins-config-volume\") pod \"23196d8b-4348-4e50-9f54-5c44c4b6b35c\" (UID: \"23196d8b-4348-4e50-9f54-5c44c4b6b35c\") " Oct 30 20:02:43.880821 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:43.880800 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23196d8b-4348-4e50-9f54-5c44c4b6b35c-plugins-config-volume" pod "23196d8b-4348-4e50-9f54-5c44c4b6b35c" (UID: "23196d8b-4348-4e50-9f54-5c44c4b6b35c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:43.882792 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:43.882763 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23196d8b-4348-4e50-9f54-5c44c4b6b35c-kube-api-access-xr8td" pod "23196d8b-4348-4e50-9f54-5c44c4b6b35c" (UID: "23196d8b-4348-4e50-9f54-5c44c4b6b35c"). InnerVolumeSpecName "kube-api-access-xr8td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:43.981888 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:43.981852 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr8td\" (UniqueName: \"kubernetes.io/projected/23196d8b-4348-4e50-9f54-5c44c4b6b35c-kube-api-access-xr8td\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:43.981888 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:43.981885 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23196d8b-4348-4e50-9f54-5c44c4b6b35c-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.431461 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:44.431416 4433 generic.go:356] "Generic (PLEG): container finished" podID="23196d8b-4348-4e50-9f54-5c44c4b6b35c" containerID="76f3117e773731c930e8598876b7bf56cc0c59a035332a34ef139c5149ff525d" exitCode=0 Oct 30 20:02:44.431685 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:44.431483 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" Oct 30 20:02:44.431685 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:44.431500 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" event={"ID":"23196d8b-4348-4e50-9f54-5c44c4b6b35c","Type":"ContainerDied","Data":"76f3117e773731c930e8598876b7bf56cc0c59a035332a34ef139c5149ff525d"} Oct 30 20:02:44.431685 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:44.431600 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7" event={"ID":"23196d8b-4348-4e50-9f54-5c44c4b6b35c","Type":"ContainerDied","Data":"e805afccf88dc7703fbcbb8d4290887f2615ba57fa5d3e957f1a8d0dfc923543"} Oct 30 20:02:44.431685 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:44.431631 4433 scope.go:122] "RemoveContainer" containerID="76f3117e773731c930e8598876b7bf56cc0c59a035332a34ef139c5149ff525d" Oct 30 20:02:44.442369 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:44.442337 4433 scope.go:122] "RemoveContainer" containerID="76f3117e773731c930e8598876b7bf56cc0c59a035332a34ef139c5149ff525d" Oct 30 20:02:44.442720 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:02:44.442697 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f3117e773731c930e8598876b7bf56cc0c59a035332a34ef139c5149ff525d\": container with ID starting with 76f3117e773731c930e8598876b7bf56cc0c59a035332a34ef139c5149ff525d not found: ID does not exist" containerID="76f3117e773731c930e8598876b7bf56cc0c59a035332a34ef139c5149ff525d" Oct 30 20:02:44.442799 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:44.442735 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f3117e773731c930e8598876b7bf56cc0c59a035332a34ef139c5149ff525d"} err="failed to get container status \"76f3117e773731c930e8598876b7bf56cc0c59a035332a34ef139c5149ff525d\": rpc error: code = NotFound desc = could not find container \"76f3117e773731c930e8598876b7bf56cc0c59a035332a34ef139c5149ff525d\": container with ID starting with 76f3117e773731c930e8598876b7bf56cc0c59a035332a34ef139c5149ff525d not found: ID does not exist" Oct 30 20:02:44.599522 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:44.599478 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7"] Oct 30 20:02:44.605886 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:44.605851 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a3f9b7-54d45c984b-52ns7"] Oct 30 20:02:44.694760 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:44.694670 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:45.222516 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:45.222442 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23196d8b-4348-4e50-9f54-5c44c4b6b35c" path="/var/lib/kubelet/pods/23196d8b-4348-4e50-9f54-5c44c4b6b35c/volumes" Oct 30 20:02:45.795300 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:45.795258 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:46.144412 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.144366 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:46.415522 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.415493 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" Oct 30 20:02:46.445775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.445701 4433 generic.go:356] "Generic (PLEG): container finished" podID="c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904" containerID="dec892ca2f8293c803cc61367f0c51afc33e8ecfb35f939fa37e9cf3cf684b3f" exitCode=0 Oct 30 20:02:46.446239 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.446192 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" Oct 30 20:02:46.446515 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.446490 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" event={"ID":"c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904","Type":"ContainerDied","Data":"dec892ca2f8293c803cc61367f0c51afc33e8ecfb35f939fa37e9cf3cf684b3f"} Oct 30 20:02:46.446621 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.446534 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb" event={"ID":"c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904","Type":"ContainerDied","Data":"e73502aa52ea42c3216c164e334e7f4a8aed9271872b457499a78aaa1b8cc110"} Oct 30 20:02:46.446621 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.446558 4433 scope.go:122] "RemoveContainer" containerID="dec892ca2f8293c803cc61367f0c51afc33e8ecfb35f939fa37e9cf3cf684b3f" Oct 30 20:02:46.448619 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.448595 4433 generic.go:356] "Generic (PLEG): container finished" podID="edac81ed-b1b2-4ddb-a2ae-bfe897991191" containerID="1d8fe6454a3d87a0b050c54a7ce2119117ecebc4d202e0746e7322c8316c8fc2" exitCode=0 Oct 30 20:02:46.448746 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.448683 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" event={"ID":"edac81ed-b1b2-4ddb-a2ae-bfe897991191","Type":"ContainerDied","Data":"1d8fe6454a3d87a0b050c54a7ce2119117ecebc4d202e0746e7322c8316c8fc2"} Oct 30 20:02:46.458766 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.458737 4433 scope.go:122] "RemoveContainer" containerID="dec892ca2f8293c803cc61367f0c51afc33e8ecfb35f939fa37e9cf3cf684b3f" Oct 30 20:02:46.459179 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:02:46.459154 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec892ca2f8293c803cc61367f0c51afc33e8ecfb35f939fa37e9cf3cf684b3f\": container with ID starting with dec892ca2f8293c803cc61367f0c51afc33e8ecfb35f939fa37e9cf3cf684b3f not found: ID does not exist" containerID="dec892ca2f8293c803cc61367f0c51afc33e8ecfb35f939fa37e9cf3cf684b3f" Oct 30 20:02:46.459297 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.459189 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec892ca2f8293c803cc61367f0c51afc33e8ecfb35f939fa37e9cf3cf684b3f"} err="failed to get container status \"dec892ca2f8293c803cc61367f0c51afc33e8ecfb35f939fa37e9cf3cf684b3f\": rpc error: code = NotFound desc = could not find container \"dec892ca2f8293c803cc61367f0c51afc33e8ecfb35f939fa37e9cf3cf684b3f\": container with ID starting with dec892ca2f8293c803cc61367f0c51afc33e8ecfb35f939fa37e9cf3cf684b3f not found: ID does not exist" Oct 30 20:02:46.508732 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.508695 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:46.523896 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.523862 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904-kube-api-access-9xg6d\" (UniqueName: \"kubernetes.io/projected/c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904-kube-api-access-9xg6d\") pod \"c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904\" (UID: \"c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904\") " Oct 30 20:02:46.524069 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.523949 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904-plugins-config-volume\") pod \"c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904\" (UID: \"c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904\") " Oct 30 20:02:46.524529 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.524500 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904-plugins-config-volume" pod "c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904" (UID: "c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:46.526653 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.526626 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904-kube-api-access-9xg6d" pod "c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904" (UID: "c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904"). InnerVolumeSpecName "kube-api-access-9xg6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:46.531397 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.531367 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" Oct 30 20:02:46.625210 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.625168 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edac81ed-b1b2-4ddb-a2ae-bfe897991191-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edac81ed-b1b2-4ddb-a2ae-bfe897991191-plugins-config-volume\") pod \"edac81ed-b1b2-4ddb-a2ae-bfe897991191\" (UID: \"edac81ed-b1b2-4ddb-a2ae-bfe897991191\") " Oct 30 20:02:46.625398 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.625285 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edac81ed-b1b2-4ddb-a2ae-bfe897991191-kube-api-access-fs569\" (UniqueName: \"kubernetes.io/projected/edac81ed-b1b2-4ddb-a2ae-bfe897991191-kube-api-access-fs569\") pod \"edac81ed-b1b2-4ddb-a2ae-bfe897991191\" (UID: \"edac81ed-b1b2-4ddb-a2ae-bfe897991191\") " Oct 30 20:02:46.625646 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.625625 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:46.625704 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.625653 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xg6d\" (UniqueName: \"kubernetes.io/projected/c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904-kube-api-access-9xg6d\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:46.625704 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.625653 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edac81ed-b1b2-4ddb-a2ae-bfe897991191-plugins-config-volume" pod "edac81ed-b1b2-4ddb-a2ae-bfe897991191" (UID: "edac81ed-b1b2-4ddb-a2ae-bfe897991191"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:46.627574 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.627539 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edac81ed-b1b2-4ddb-a2ae-bfe897991191-kube-api-access-fs569" pod "edac81ed-b1b2-4ddb-a2ae-bfe897991191" (UID: "edac81ed-b1b2-4ddb-a2ae-bfe897991191"). InnerVolumeSpecName "kube-api-access-fs569". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:46.727347 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.727245 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edac81ed-b1b2-4ddb-a2ae-bfe897991191-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:46.727347 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.727285 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs569\" (UniqueName: \"kubernetes.io/projected/edac81ed-b1b2-4ddb-a2ae-bfe897991191-kube-api-access-fs569\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:02:46.866958 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.866919 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb"] Oct 30 20:02:46.872929 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.872894 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ada8d7-7bc6d59b78-ppbmb"] Oct 30 20:02:46.890726 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:46.890695 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:47.124893 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:47.124862 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:47.217125 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:47.217091 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904" path="/var/lib/kubelet/pods/c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904/volumes" Oct 30 20:02:47.351117 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:47.351080 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:47.456698 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:47.456599 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" Oct 30 20:02:47.456698 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:47.456614 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt" event={"ID":"edac81ed-b1b2-4ddb-a2ae-bfe897991191","Type":"ContainerDied","Data":"6330b82928a3c9ebe0e2ce2699fb3abd07ec8ceb08f35786ae2cd238fdc0b125"} Oct 30 20:02:47.456698 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:47.456684 4433 scope.go:122] "RemoveContainer" containerID="1d8fe6454a3d87a0b050c54a7ce2119117ecebc4d202e0746e7322c8316c8fc2" Oct 30 20:02:47.582389 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:47.582351 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:47.624634 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:47.624591 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt"] Oct 30 20:02:47.671434 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:47.671397 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ada8d7-989f9765f-lfcdt"] Oct 30 20:02:47.803636 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:47.803527 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:48.306763 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:48.306723 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:49.213155 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:49.213116 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edac81ed-b1b2-4ddb-a2ae-bfe897991191" path="/var/lib/kubelet/pods/edac81ed-b1b2-4ddb-a2ae-bfe897991191/volumes" Oct 30 20:02:50.455488 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:50.455458 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:50.677462 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:50.677428 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:51.420905 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:51.420869 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:51.870359 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:51.870295 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:52.610012 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:52.609936 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:52.989824 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:52.989730 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:53.371658 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:02:53.371611 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:02:53.523144 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:53.523109 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:53.782910 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:53.782827 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:54.031430 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:54.031393 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:55.335036 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:55.334963 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:55.573781 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:55.573706 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:55.806428 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:55.806289 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v"] Oct 30 20:02:55.806624 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:55.806598 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" podUID="75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" containerName="istio-proxy" containerID="cri-o://0ee053c921b54468e37d4194dd846c5ef828ddd9a783d6248cfc6d5fc984b7d1" gracePeriod=30 Oct 30 20:02:55.972467 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:55.972416 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:57.365954 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:57.365916 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:57.826498 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:57.826460 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:59.596205 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:59.596158 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:59.837792 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:02:59.837754 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:01.075426 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.075346 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 20:03:01.182332 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.182282 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-podinfo\") pod \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " Oct 30 20:03:01.182332 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.182329 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-token\" (UniqueName: \"kubernetes.io/projected/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-token\") pod \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " Oct 30 20:03:01.182618 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.182362 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-data\") pod \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " Oct 30 20:03:01.182618 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.182395 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-workload-certs\") pod \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " Oct 30 20:03:01.182618 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.182441 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-envoy\") pod \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " Oct 30 20:03:01.182618 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.182589 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-workload-socket\") pod \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " Oct 30 20:03:01.182825 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.182651 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-credential-socket\") pod \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " Oct 30 20:03:01.182825 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.182706 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istiod-ca-cert\") pod \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " Oct 30 20:03:01.182825 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.182761 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-workload-certs" pod "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" (UID: "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:01.182825 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.182798 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-kube-api-access-v2d4k\" (UniqueName: \"kubernetes.io/projected/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-kube-api-access-v2d4k\") pod \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\" (UID: \"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb\") " Oct 30 20:03:01.183089 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.182954 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-credential-socket" pod "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" (UID: "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:01.183089 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.182770 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-data" pod "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" (UID: "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:01.183204 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.183110 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-workload-socket" pod "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" (UID: "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:01.183204 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.183116 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istiod-ca-cert" pod "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" (UID: "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:01.183312 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.183279 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:03:01.183312 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.183295 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:03:01.183312 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.183309 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:03:01.183470 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.183319 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:03:01.183470 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.183327 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:03:01.184915 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.184885 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-token" pod "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" (UID: "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:01.185068 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.184964 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-kube-api-access-v2d4k" pod "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" (UID: "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb"). InnerVolumeSpecName "kube-api-access-v2d4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:01.185207 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.185186 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-envoy" pod "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" (UID: "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:01.185283 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.185208 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-podinfo" pod "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" (UID: "75c1ea2e-e33e-4828-abd3-266fcc1a8dbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:01.284872 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.284827 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:03:01.284872 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.284871 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:03:01.285087 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.284886 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:03:01.285087 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.284900 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2d4k\" (UniqueName: \"kubernetes.io/projected/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb-kube-api-access-v2d4k\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:03:01.541744 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.541642 4433 generic.go:356] "Generic (PLEG): container finished" podID="75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" containerID="0ee053c921b54468e37d4194dd846c5ef828ddd9a783d6248cfc6d5fc984b7d1" exitCode=0 Oct 30 20:03:01.541744 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.541701 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" Oct 30 20:03:01.542013 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.541703 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" event={"ID":"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb","Type":"ContainerDied","Data":"0ee053c921b54468e37d4194dd846c5ef828ddd9a783d6248cfc6d5fc984b7d1"} Oct 30 20:03:01.542013 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.541833 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v" event={"ID":"75c1ea2e-e33e-4828-abd3-266fcc1a8dbb","Type":"ContainerDied","Data":"fe898419564f6e7ccac4e7af72f5cdc3510d55b2aa41183e60904afdee2d7cc5"} Oct 30 20:03:01.542013 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.541863 4433 scope.go:122] "RemoveContainer" containerID="0ee053c921b54468e37d4194dd846c5ef828ddd9a783d6248cfc6d5fc984b7d1" Oct 30 20:03:01.551166 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.551142 4433 scope.go:122] "RemoveContainer" containerID="0ee053c921b54468e37d4194dd846c5ef828ddd9a783d6248cfc6d5fc984b7d1" Oct 30 20:03:01.551531 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:03:01.551504 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee053c921b54468e37d4194dd846c5ef828ddd9a783d6248cfc6d5fc984b7d1\": container with ID starting with 0ee053c921b54468e37d4194dd846c5ef828ddd9a783d6248cfc6d5fc984b7d1 not found: ID does not exist" containerID="0ee053c921b54468e37d4194dd846c5ef828ddd9a783d6248cfc6d5fc984b7d1" Oct 30 20:03:01.551588 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.551542 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee053c921b54468e37d4194dd846c5ef828ddd9a783d6248cfc6d5fc984b7d1"} err="failed to get container status \"0ee053c921b54468e37d4194dd846c5ef828ddd9a783d6248cfc6d5fc984b7d1\": rpc error: code = NotFound desc = could not find container \"0ee053c921b54468e37d4194dd846c5ef828ddd9a783d6248cfc6d5fc984b7d1\": container with ID starting with 0ee053c921b54468e37d4194dd846c5ef828ddd9a783d6248cfc6d5fc984b7d1 not found: ID does not exist" Oct 30 20:03:01.710652 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.710612 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v"] Oct 30 20:03:01.715739 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:01.715703 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-xp64v"] Oct 30 20:03:02.058348 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:02.058314 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:03.213546 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:03.213496 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" path="/var/lib/kubelet/pods/75c1ea2e-e33e-4828-abd3-266fcc1a8dbb/volumes" Oct 30 20:03:04.017872 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:04.017823 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:04.241323 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:04.241288 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:06.676367 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:06.676278 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:11.459813 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:11.459770 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:16.071021 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:16.070943 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:20.764615 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:20.764577 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:23.369013 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:03:23.368939 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:03:25.351107 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:25.351066 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:29.989627 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:29.989583 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:31.988852 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:31.988814 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:34.774177 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:34.774138 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:39.450240 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:39.450199 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:44.148278 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:44.148241 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:48.755998 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:48.755936 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:51.758753 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:51.758715 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:53.356068 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:53.356035 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:53.367196 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:03:53.367167 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:03:55.670363 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:55.670324 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:57.781656 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:57.781618 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:58.151459 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:03:58.151404 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:01.975219 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:01.975166 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" podUID="d17eaea73d7bf32a8a9bb9e63451f939" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:02.783525 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:02.783489 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:07.379149 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:07.379106 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:12.066721 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:12.066681 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:16.725712 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:16.725674 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:20.704915 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:20.704882 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:21.538323 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:21.538282 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:23.368761 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:04:23.368723 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:04:26.084299 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:26.084254 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:30.666371 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:30.666312 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:32.401028 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:32.400820 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:35.425314 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:35.425185 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:40.116083 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:40.116044 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:44.821401 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:44.821365 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:46.818684 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:46.818643 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:49.652809 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:49.652768 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:52.117294 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:52.117249 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:53.367556 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:04:53.367514 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:04:54.551711 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:54.551680 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:56.015876 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:56.015835 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:59.144191 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:59.144120 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:59.384039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:04:59.383975 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:03.924762 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:03.924723 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:08.594243 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:08.594211 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:12.712686 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:12.712651 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:14.238771 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:14.238731 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:14.480068 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:14.480029 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:14.715223 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:14.715185 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:19.544109 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:19.544068 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:23.367034 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:05:23.366970 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:05:24.140094 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:24.140058 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:25.251664 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:25.251626 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:25.984709 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:25.984669 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv"] Oct 30 20:05:25.984959 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:25.984917 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" podUID="bc78983a-aa83-48e3-b0c3-95148da6dfc1" containerName="discovery" containerID="cri-o://fc51aea267f4fd3108796d990854fd6918dd14c543cde64583895b3cee7ea185" gracePeriod=30 Oct 30 20:05:26.238062 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.237971 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 20:05:26.320172 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.320133 4433 generic.go:356] "Generic (PLEG): container finished" podID="bc78983a-aa83-48e3-b0c3-95148da6dfc1" containerID="fc51aea267f4fd3108796d990854fd6918dd14c543cde64583895b3cee7ea185" exitCode=0 Oct 30 20:05:26.320674 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.320189 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" event={"ID":"bc78983a-aa83-48e3-b0c3-95148da6dfc1","Type":"ContainerDied","Data":"fc51aea267f4fd3108796d990854fd6918dd14c543cde64583895b3cee7ea185"} Oct 30 20:05:26.320674 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.320241 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" event={"ID":"bc78983a-aa83-48e3-b0c3-95148da6dfc1","Type":"ContainerDied","Data":"d9e6db86fd0118ea67f0f58cb53a5e0b9e4e2fb1b965fe5d700c70ea58a19bed"} Oct 30 20:05:26.320674 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.320257 4433 scope.go:122] "RemoveContainer" containerID="fc51aea267f4fd3108796d990854fd6918dd14c543cde64583895b3cee7ea185" Oct 30 20:05:26.320674 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.320242 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv" Oct 30 20:05:26.324555 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.324529 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc78983a-aa83-48e3-b0c3-95148da6dfc1-kube-api-access-v758w\" (UniqueName: \"kubernetes.io/projected/bc78983a-aa83-48e3-b0c3-95148da6dfc1-kube-api-access-v758w\") pod \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " Oct 30 20:05:26.324697 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.324563 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-csr-ca-configmap\") pod \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " Oct 30 20:05:26.324697 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.324622 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-cacerts\" (UniqueName: \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-cacerts\") pod \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " Oct 30 20:05:26.324697 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.324670 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-kubeconfig\") pod \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " Oct 30 20:05:26.324857 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.324720 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-token\" (UniqueName: \"kubernetes.io/projected/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-token\") pod \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " Oct 30 20:05:26.324857 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.324752 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-csr-dns-cert\") pod \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " Oct 30 20:05:26.324857 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.324786 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc78983a-aa83-48e3-b0c3-95148da6dfc1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc78983a-aa83-48e3-b0c3-95148da6dfc1-local-certs\") pod \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\" (UID: \"bc78983a-aa83-48e3-b0c3-95148da6dfc1\") " Oct 30 20:05:26.325218 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.325187 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-csr-ca-configmap" pod "bc78983a-aa83-48e3-b0c3-95148da6dfc1" (UID: "bc78983a-aa83-48e3-b0c3-95148da6dfc1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:26.327200 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.327173 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc78983a-aa83-48e3-b0c3-95148da6dfc1-kube-api-access-v758w" pod "bc78983a-aa83-48e3-b0c3-95148da6dfc1" (UID: "bc78983a-aa83-48e3-b0c3-95148da6dfc1"). InnerVolumeSpecName "kube-api-access-v758w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:26.327315 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.327260 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-csr-dns-cert" pod "bc78983a-aa83-48e3-b0c3-95148da6dfc1" (UID: "bc78983a-aa83-48e3-b0c3-95148da6dfc1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:26.327665 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.327641 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-kubeconfig" pod "bc78983a-aa83-48e3-b0c3-95148da6dfc1" (UID: "bc78983a-aa83-48e3-b0c3-95148da6dfc1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:26.327754 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.327715 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc78983a-aa83-48e3-b0c3-95148da6dfc1-local-certs" pod "bc78983a-aa83-48e3-b0c3-95148da6dfc1" (UID: "bc78983a-aa83-48e3-b0c3-95148da6dfc1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:05:26.327799 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.327754 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-cacerts" pod "bc78983a-aa83-48e3-b0c3-95148da6dfc1" (UID: "bc78983a-aa83-48e3-b0c3-95148da6dfc1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:26.327834 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.327823 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-token" pod "bc78983a-aa83-48e3-b0c3-95148da6dfc1" (UID: "bc78983a-aa83-48e3-b0c3-95148da6dfc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:26.349598 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.349578 4433 scope.go:122] "RemoveContainer" containerID="fc51aea267f4fd3108796d990854fd6918dd14c543cde64583895b3cee7ea185" Oct 30 20:05:26.349897 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:05:26.349878 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc51aea267f4fd3108796d990854fd6918dd14c543cde64583895b3cee7ea185\": container with ID starting with fc51aea267f4fd3108796d990854fd6918dd14c543cde64583895b3cee7ea185 not found: ID does not exist" containerID="fc51aea267f4fd3108796d990854fd6918dd14c543cde64583895b3cee7ea185" Oct 30 20:05:26.349969 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.349908 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc51aea267f4fd3108796d990854fd6918dd14c543cde64583895b3cee7ea185"} err="failed to get container status \"fc51aea267f4fd3108796d990854fd6918dd14c543cde64583895b3cee7ea185\": rpc error: code = NotFound desc = could not find container \"fc51aea267f4fd3108796d990854fd6918dd14c543cde64583895b3cee7ea185\": container with ID starting with fc51aea267f4fd3108796d990854fd6918dd14c543cde64583895b3cee7ea185 not found: ID does not exist" Oct 30 20:05:26.426286 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.426249 4433 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc78983a-aa83-48e3-b0c3-95148da6dfc1-local-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:05:26.426286 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.426286 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v758w\" (UniqueName: \"kubernetes.io/projected/bc78983a-aa83-48e3-b0c3-95148da6dfc1-kube-api-access-v758w\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:05:26.426471 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.426303 4433 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-csr-ca-configmap\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:05:26.426471 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.426319 4433 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-cacerts\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:05:26.426471 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.426333 4433 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-kubeconfig\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:05:26.426471 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.426356 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:05:26.426471 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.426370 4433 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bc78983a-aa83-48e3-b0c3-95148da6dfc1-istio-csr-dns-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:05:26.741000 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.740951 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv"] Oct 30 20:05:26.743791 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:26.743759 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pxwxv"] Oct 30 20:05:27.212432 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:27.212393 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc78983a-aa83-48e3-b0c3-95148da6dfc1" path="/var/lib/kubelet/pods/bc78983a-aa83-48e3-b0c3-95148da6dfc1/volumes" Oct 30 20:05:28.837927 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:28.837888 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:31.554922 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:31.554891 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:31.800756 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:31.800720 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:32.023057 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:32.023022 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:32.683428 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:32.683393 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:33.601690 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:33.601653 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:38.477549 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:38.477512 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:43.109668 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:43.109631 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:45.038323 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:45.038286 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:47.236024 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:47.235974 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:48.762613 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:48.762561 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:48.982298 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:48.982266 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:49.627365 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:49.627320 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:53.366530 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:05:53.366496 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:05:53.775741 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:53.775655 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:56.105061 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:56.105025 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:56.334845 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:56.334805 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:56.557417 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:56.557328 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:56.929744 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:56.929707 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:58.686416 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:05:58.686384 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:01.799722 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:01.799684 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m"] Oct 30 20:06:01.848931 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:01.848895 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c"] Oct 30 20:06:02.041615 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:02.041580 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:02.451122 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:02.451079 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:03.558410 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:03.558370 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:08.134236 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:08.134135 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:08.806078 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:08.806042 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd"] Oct 30 20:06:08.912163 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:08.912122 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t"] Oct 30 20:06:09.251529 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:09.251490 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:09.468571 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:09.468536 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:13.806511 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:13.806448 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" podUID="f54b26b9-651b-4ba4-82a4-57819b4f02b4" containerName="payload-processing" containerID="cri-o://aaddbbf3c4840d1108cad0187494b149c59a80b614ee09146440a08d8c5dc9d2" gracePeriod=25 Oct 30 20:06:13.912478 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:13.912428 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" podUID="5ecee9ae-a839-4edc-b06b-cc014cb7e7ff" containerName="payload-pre-processing" containerID="cri-o://e328f93251d91e0deb76a6735b7f0e00b9dc38ab1732c2a819946ad63622431f" gracePeriod=25 Oct 30 20:06:16.820465 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:16.820417 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" podUID="6c890add-9063-4067-bb5e-65324cf99b3d" containerName="main" containerID="cri-o://1875ecf370b68a6a735ce86d489ffd829a18b283b23b706b92d6773b3e75b123" gracePeriod=45 Oct 30 20:06:16.869068 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:16.869019 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" podUID="0d290229-7e04-47a2-8c0a-c18988cd07f0" containerName="main" containerID="cri-o://477cfe62a56919fc46f5a621408f63cc374fd884be57474c89a2a05cd8b287ac" gracePeriod=45 Oct 30 20:06:17.128806 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.128777 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 20:06:17.152336 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.152313 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.246807 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-tmp-dir\") pod \"0d290229-7e04-47a2-8c0a-c18988cd07f0\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.247169 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-dshm\") pod \"6c890add-9063-4067-bb5e-65324cf99b3d\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.247317 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-home\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-home\") pod \"6c890add-9063-4067-bb5e-65324cf99b3d\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.247536 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d290229-7e04-47a2-8c0a-c18988cd07f0-kube-api-access-cclq4\" (UniqueName: \"kubernetes.io/projected/0d290229-7e04-47a2-8c0a-c18988cd07f0-kube-api-access-cclq4\") pod \"0d290229-7e04-47a2-8c0a-c18988cd07f0\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.247624 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-home\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-home\") pod \"0d290229-7e04-47a2-8c0a-c18988cd07f0\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.247714 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d290229-7e04-47a2-8c0a-c18988cd07f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d290229-7e04-47a2-8c0a-c18988cd07f0-tls-certs\") pod \"0d290229-7e04-47a2-8c0a-c18988cd07f0\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.247807 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c890add-9063-4067-bb5e-65324cf99b3d-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c890add-9063-4067-bb5e-65324cf99b3d-tls-certs\") pod \"6c890add-9063-4067-bb5e-65324cf99b3d\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.247871 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c890add-9063-4067-bb5e-65324cf99b3d-kube-api-access-nvnnx\" (UniqueName: \"kubernetes.io/projected/6c890add-9063-4067-bb5e-65324cf99b3d-kube-api-access-nvnnx\") pod \"6c890add-9063-4067-bb5e-65324cf99b3d\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.247964 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-model-cache\") pod \"0d290229-7e04-47a2-8c0a-c18988cd07f0\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.248042 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-model-cache\") pod \"6c890add-9063-4067-bb5e-65324cf99b3d\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.248109 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-tmp-dir\") pod \"6c890add-9063-4067-bb5e-65324cf99b3d\" (UID: \"6c890add-9063-4067-bb5e-65324cf99b3d\") " Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.248174 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-dshm\") pod \"0d290229-7e04-47a2-8c0a-c18988cd07f0\" (UID: \"0d290229-7e04-47a2-8c0a-c18988cd07f0\") " Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.251094 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-model-cache" pod "0d290229-7e04-47a2-8c0a-c18988cd07f0" (UID: "0d290229-7e04-47a2-8c0a-c18988cd07f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.251886 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d290229-7e04-47a2-8c0a-c18988cd07f0-tls-certs" pod "0d290229-7e04-47a2-8c0a-c18988cd07f0" (UID: "0d290229-7e04-47a2-8c0a-c18988cd07f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.252242 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-tmp-dir" pod "0d290229-7e04-47a2-8c0a-c18988cd07f0" (UID: "0d290229-7e04-47a2-8c0a-c18988cd07f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.254317 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c890add-9063-4067-bb5e-65324cf99b3d-tls-certs" pod "6c890add-9063-4067-bb5e-65324cf99b3d" (UID: "6c890add-9063-4067-bb5e-65324cf99b3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.254862 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-dshm" pod "6c890add-9063-4067-bb5e-65324cf99b3d" (UID: "6c890add-9063-4067-bb5e-65324cf99b3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.255114 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-home" pod "6c890add-9063-4067-bb5e-65324cf99b3d" (UID: "6c890add-9063-4067-bb5e-65324cf99b3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.256672 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c890add-9063-4067-bb5e-65324cf99b3d-kube-api-access-nvnnx" pod "6c890add-9063-4067-bb5e-65324cf99b3d" (UID: "6c890add-9063-4067-bb5e-65324cf99b3d"). InnerVolumeSpecName "kube-api-access-nvnnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:17.256999 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.256891 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-home" pod "0d290229-7e04-47a2-8c0a-c18988cd07f0" (UID: "0d290229-7e04-47a2-8c0a-c18988cd07f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:17.258124 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.257156 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-model-cache" pod "6c890add-9063-4067-bb5e-65324cf99b3d" (UID: "6c890add-9063-4067-bb5e-65324cf99b3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:17.258124 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.257379 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d290229-7e04-47a2-8c0a-c18988cd07f0-kube-api-access-cclq4" pod "0d290229-7e04-47a2-8c0a-c18988cd07f0" (UID: "0d290229-7e04-47a2-8c0a-c18988cd07f0"). InnerVolumeSpecName "kube-api-access-cclq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:17.258577 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.258545 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-tmp-dir\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.258577 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.258580 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-dshm\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.258768 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.258594 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-home\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.258768 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.258617 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cclq4\" (UniqueName: \"kubernetes.io/projected/0d290229-7e04-47a2-8c0a-c18988cd07f0-kube-api-access-cclq4\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.258768 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.258631 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-home\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.258768 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.258644 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d290229-7e04-47a2-8c0a-c18988cd07f0-tls-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.258768 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.258657 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c890add-9063-4067-bb5e-65324cf99b3d-tls-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.258768 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.258674 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvnnx\" (UniqueName: \"kubernetes.io/projected/6c890add-9063-4067-bb5e-65324cf99b3d-kube-api-access-nvnnx\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.258768 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.258688 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-model-cache\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.258768 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.258703 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-model-cache\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.259324 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.259017 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-tmp-dir" pod "6c890add-9063-4067-bb5e-65324cf99b3d" (UID: "6c890add-9063-4067-bb5e-65324cf99b3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:17.261625 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.261593 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-dshm" pod "0d290229-7e04-47a2-8c0a-c18988cd07f0" (UID: "0d290229-7e04-47a2-8c0a-c18988cd07f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:17.359804 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.359694 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d290229-7e04-47a2-8c0a-c18988cd07f0-dshm\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.359804 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.359732 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c890add-9063-4067-bb5e-65324cf99b3d-tmp-dir\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.586286 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.586249 4433 generic.go:356] "Generic (PLEG): container finished" podID="0d290229-7e04-47a2-8c0a-c18988cd07f0" containerID="477cfe62a56919fc46f5a621408f63cc374fd884be57474c89a2a05cd8b287ac" exitCode=0 Oct 30 20:06:17.586500 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.586311 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" Oct 30 20:06:17.586500 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.586341 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" event={"ID":"0d290229-7e04-47a2-8c0a-c18988cd07f0","Type":"ContainerDied","Data":"477cfe62a56919fc46f5a621408f63cc374fd884be57474c89a2a05cd8b287ac"} Oct 30 20:06:17.586500 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.586385 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c" event={"ID":"0d290229-7e04-47a2-8c0a-c18988cd07f0","Type":"ContainerDied","Data":"07f51c8816638ab2d1b7f80dd4fc211a24da9c80c05fc082396c75e16a71ef25"} Oct 30 20:06:17.586500 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.586406 4433 scope.go:122] "RemoveContainer" containerID="477cfe62a56919fc46f5a621408f63cc374fd884be57474c89a2a05cd8b287ac" Oct 30 20:06:17.587817 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.587793 4433 generic.go:356] "Generic (PLEG): container finished" podID="6c890add-9063-4067-bb5e-65324cf99b3d" containerID="1875ecf370b68a6a735ce86d489ffd829a18b283b23b706b92d6773b3e75b123" exitCode=0 Oct 30 20:06:17.587936 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.587848 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" Oct 30 20:06:17.587936 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.587894 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" event={"ID":"6c890add-9063-4067-bb5e-65324cf99b3d","Type":"ContainerDied","Data":"1875ecf370b68a6a735ce86d489ffd829a18b283b23b706b92d6773b3e75b123"} Oct 30 20:06:17.587936 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.587918 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m" event={"ID":"6c890add-9063-4067-bb5e-65324cf99b3d","Type":"ContainerDied","Data":"d030cd067d51e9b83df9dc93ffd9499e7176c2338c8d3da10ce3ffb56f9ad117"} Oct 30 20:06:17.595274 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.595248 4433 scope.go:122] "RemoveContainer" containerID="477cfe62a56919fc46f5a621408f63cc374fd884be57474c89a2a05cd8b287ac" Oct 30 20:06:17.595541 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:06:17.595521 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"477cfe62a56919fc46f5a621408f63cc374fd884be57474c89a2a05cd8b287ac\": container with ID starting with 477cfe62a56919fc46f5a621408f63cc374fd884be57474c89a2a05cd8b287ac not found: ID does not exist" containerID="477cfe62a56919fc46f5a621408f63cc374fd884be57474c89a2a05cd8b287ac" Oct 30 20:06:17.595631 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.595554 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"477cfe62a56919fc46f5a621408f63cc374fd884be57474c89a2a05cd8b287ac"} err="failed to get container status \"477cfe62a56919fc46f5a621408f63cc374fd884be57474c89a2a05cd8b287ac\": rpc error: code = NotFound desc = could not find container \"477cfe62a56919fc46f5a621408f63cc374fd884be57474c89a2a05cd8b287ac\": container with ID starting with 477cfe62a56919fc46f5a621408f63cc374fd884be57474c89a2a05cd8b287ac not found: ID does not exist" Oct 30 20:06:17.595631 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.595579 4433 scope.go:122] "RemoveContainer" containerID="1875ecf370b68a6a735ce86d489ffd829a18b283b23b706b92d6773b3e75b123" Oct 30 20:06:17.604181 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.604164 4433 scope.go:122] "RemoveContainer" containerID="1875ecf370b68a6a735ce86d489ffd829a18b283b23b706b92d6773b3e75b123" Oct 30 20:06:17.604444 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:06:17.604426 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1875ecf370b68a6a735ce86d489ffd829a18b283b23b706b92d6773b3e75b123\": container with ID starting with 1875ecf370b68a6a735ce86d489ffd829a18b283b23b706b92d6773b3e75b123 not found: ID does not exist" containerID="1875ecf370b68a6a735ce86d489ffd829a18b283b23b706b92d6773b3e75b123" Oct 30 20:06:17.604497 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.604453 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1875ecf370b68a6a735ce86d489ffd829a18b283b23b706b92d6773b3e75b123"} err="failed to get container status \"1875ecf370b68a6a735ce86d489ffd829a18b283b23b706b92d6773b3e75b123\": rpc error: code = NotFound desc = could not find container \"1875ecf370b68a6a735ce86d489ffd829a18b283b23b706b92d6773b3e75b123\": container with ID starting with 1875ecf370b68a6a735ce86d489ffd829a18b283b23b706b92d6773b3e75b123 not found: ID does not exist" Oct 30 20:06:17.748249 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.748155 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c"] Oct 30 20:06:17.797488 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.797458 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/simulator-w4-b5178f-kserve-7559d7c57d-mxg5c"] Oct 30 20:06:17.859334 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.859304 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m"] Oct 30 20:06:17.864965 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:17.864939 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/distinct-w4-b5178f-kserve-5f6cd664c4-6fl2m"] Oct 30 20:06:19.212410 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:19.212372 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d290229-7e04-47a2-8c0a-c18988cd07f0" path="/var/lib/kubelet/pods/0d290229-7e04-47a2-8c0a-c18988cd07f0/volumes" Oct 30 20:06:19.213025 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:19.213004 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c890add-9063-4067-bb5e-65324cf99b3d" path="/var/lib/kubelet/pods/6c890add-9063-4067-bb5e-65324cf99b3d/volumes" Oct 30 20:06:21.536053 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:21.536014 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" podUID="5ecee9ae-a839-4edc-b06b-cc014cb7e7ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:06:22.541098 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:22.541055 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" podUID="f54b26b9-651b-4ba4-82a4-57819b4f02b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:06:23.367676 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:06:23.367558 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:06:27.503873 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:27.503834 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s"] Oct 30 20:06:27.640255 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:27.640196 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" containerID="cri-o://aecbb010c96896016077a9a3ba4bd42784181015e31ca205b03b6d19fcbb5b2f" gracePeriod=30 Oct 30 20:06:27.691931 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:27.691887 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:27.901353 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:27.901325 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 20:06:27.962431 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:27.962389 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0e2f76e-309b-46e6-8ee8-478c0def15fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e2f76e-309b-46e6-8ee8-478c0def15fc-openshift-service-ca\") pod \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " Oct 30 20:06:27.962626 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:27.962524 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0e2f76e-309b-46e6-8ee8-478c0def15fc-kube-api-access-gmprz\" (UniqueName: \"kubernetes.io/projected/b0e2f76e-309b-46e6-8ee8-478c0def15fc-kube-api-access-gmprz\") pod \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " Oct 30 20:06:27.962626 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:27.962568 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-maas-api-tls\") pod \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " Oct 30 20:06:27.962764 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:27.962647 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-metrics-certs\") pod \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\" (UID: \"b0e2f76e-309b-46e6-8ee8-478c0def15fc\") " Oct 30 20:06:27.962822 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:27.962800 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0e2f76e-309b-46e6-8ee8-478c0def15fc-openshift-service-ca" pod "b0e2f76e-309b-46e6-8ee8-478c0def15fc" (UID: "b0e2f76e-309b-46e6-8ee8-478c0def15fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:27.963015 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:27.962974 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e2f76e-309b-46e6-8ee8-478c0def15fc-openshift-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:27.964790 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:27.964757 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-metrics-certs" pod "b0e2f76e-309b-46e6-8ee8-478c0def15fc" (UID: "b0e2f76e-309b-46e6-8ee8-478c0def15fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:27.965256 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:27.965229 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-maas-api-tls" pod "b0e2f76e-309b-46e6-8ee8-478c0def15fc" (UID: "b0e2f76e-309b-46e6-8ee8-478c0def15fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:27.965368 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:27.965285 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e2f76e-309b-46e6-8ee8-478c0def15fc-kube-api-access-gmprz" pod "b0e2f76e-309b-46e6-8ee8-478c0def15fc" (UID: "b0e2f76e-309b-46e6-8ee8-478c0def15fc"). InnerVolumeSpecName "kube-api-access-gmprz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:28.064344 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:28.064315 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmprz\" (UniqueName: \"kubernetes.io/projected/b0e2f76e-309b-46e6-8ee8-478c0def15fc-kube-api-access-gmprz\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:28.064344 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:28.064344 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-maas-api-tls\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:28.064542 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:28.064355 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0e2f76e-309b-46e6-8ee8-478c0def15fc-metrics-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:28.648303 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:28.648264 4433 generic.go:356] "Generic (PLEG): container finished" podID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerID="aecbb010c96896016077a9a3ba4bd42784181015e31ca205b03b6d19fcbb5b2f" exitCode=0 Oct 30 20:06:28.648883 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:28.648334 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" Oct 30 20:06:28.648883 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:28.648358 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" event={"ID":"b0e2f76e-309b-46e6-8ee8-478c0def15fc","Type":"ContainerDied","Data":"aecbb010c96896016077a9a3ba4bd42784181015e31ca205b03b6d19fcbb5b2f"} Oct 30 20:06:28.648883 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:28.648414 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s" event={"ID":"b0e2f76e-309b-46e6-8ee8-478c0def15fc","Type":"ContainerDied","Data":"816c6104095d9b862c8ba6f7afdbffa2bc3e73bcb11107930904b553bd6c83f8"} Oct 30 20:06:28.648883 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:28.648440 4433 scope.go:122] "RemoveContainer" containerID="aecbb010c96896016077a9a3ba4bd42784181015e31ca205b03b6d19fcbb5b2f" Oct 30 20:06:28.659033 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:28.659012 4433 scope.go:122] "RemoveContainer" containerID="153743b6efcb900ca15ddae35e4c609df2a619e76e8fa8e2c3910f96cf9f6f0d" Oct 30 20:06:28.667565 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:28.667542 4433 scope.go:122] "RemoveContainer" containerID="aecbb010c96896016077a9a3ba4bd42784181015e31ca205b03b6d19fcbb5b2f" Oct 30 20:06:28.667841 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:06:28.667822 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aecbb010c96896016077a9a3ba4bd42784181015e31ca205b03b6d19fcbb5b2f\": container with ID starting with aecbb010c96896016077a9a3ba4bd42784181015e31ca205b03b6d19fcbb5b2f not found: ID does not exist" containerID="aecbb010c96896016077a9a3ba4bd42784181015e31ca205b03b6d19fcbb5b2f" Oct 30 20:06:28.667931 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:28.667851 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aecbb010c96896016077a9a3ba4bd42784181015e31ca205b03b6d19fcbb5b2f"} err="failed to get container status \"aecbb010c96896016077a9a3ba4bd42784181015e31ca205b03b6d19fcbb5b2f\": rpc error: code = NotFound desc = could not find container \"aecbb010c96896016077a9a3ba4bd42784181015e31ca205b03b6d19fcbb5b2f\": container with ID starting with aecbb010c96896016077a9a3ba4bd42784181015e31ca205b03b6d19fcbb5b2f not found: ID does not exist" Oct 30 20:06:28.667931 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:28.667870 4433 scope.go:122] "RemoveContainer" containerID="153743b6efcb900ca15ddae35e4c609df2a619e76e8fa8e2c3910f96cf9f6f0d" Oct 30 20:06:28.668113 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:06:28.668093 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153743b6efcb900ca15ddae35e4c609df2a619e76e8fa8e2c3910f96cf9f6f0d\": container with ID starting with 153743b6efcb900ca15ddae35e4c609df2a619e76e8fa8e2c3910f96cf9f6f0d not found: ID does not exist" containerID="153743b6efcb900ca15ddae35e4c609df2a619e76e8fa8e2c3910f96cf9f6f0d" Oct 30 20:06:28.668154 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:28.668120 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153743b6efcb900ca15ddae35e4c609df2a619e76e8fa8e2c3910f96cf9f6f0d"} err="failed to get container status \"153743b6efcb900ca15ddae35e4c609df2a619e76e8fa8e2c3910f96cf9f6f0d\": rpc error: code = NotFound desc = could not find container \"153743b6efcb900ca15ddae35e4c609df2a619e76e8fa8e2c3910f96cf9f6f0d\": container with ID starting with 153743b6efcb900ca15ddae35e4c609df2a619e76e8fa8e2c3910f96cf9f6f0d not found: ID does not exist" Oct 30 20:06:28.811674 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:28.811618 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s"] Oct 30 20:06:28.860595 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:28.860558 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b5178f-6d5c4c596c-96r6s"] Oct 30 20:06:29.172851 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:29.172804 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:29.212189 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:29.212156 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" path="/var/lib/kubelet/pods/b0e2f76e-309b-46e6-8ee8-478c0def15fc/volumes" Oct 30 20:06:29.459788 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:29.459709 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:30.121566 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:30.121531 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:31.535608 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:31.535565 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" podUID="5ecee9ae-a839-4edc-b06b-cc014cb7e7ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:06:32.540915 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:32.540868 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" podUID="f54b26b9-651b-4ba4-82a4-57819b4f02b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:06:33.018243 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:33.018205 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:34.057380 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.057348 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" Oct 30 20:06:34.125480 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.125437 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54b26b9-651b-4ba4-82a4-57819b4f02b4-kube-api-access-tjnhb\" (UniqueName: \"kubernetes.io/projected/f54b26b9-651b-4ba4-82a4-57819b4f02b4-kube-api-access-tjnhb\") pod \"f54b26b9-651b-4ba4-82a4-57819b4f02b4\" (UID: \"f54b26b9-651b-4ba4-82a4-57819b4f02b4\") " Oct 30 20:06:34.125680 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.125519 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f54b26b9-651b-4ba4-82a4-57819b4f02b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f54b26b9-651b-4ba4-82a4-57819b4f02b4-plugins-config-volume\") pod \"f54b26b9-651b-4ba4-82a4-57819b4f02b4\" (UID: \"f54b26b9-651b-4ba4-82a4-57819b4f02b4\") " Oct 30 20:06:34.126447 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.126413 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54b26b9-651b-4ba4-82a4-57819b4f02b4-plugins-config-volume" pod "f54b26b9-651b-4ba4-82a4-57819b4f02b4" (UID: "f54b26b9-651b-4ba4-82a4-57819b4f02b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:34.128724 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.128688 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54b26b9-651b-4ba4-82a4-57819b4f02b4-kube-api-access-tjnhb" pod "f54b26b9-651b-4ba4-82a4-57819b4f02b4" (UID: "f54b26b9-651b-4ba4-82a4-57819b4f02b4"). InnerVolumeSpecName "kube-api-access-tjnhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:34.156565 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.156540 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" Oct 30 20:06:34.226830 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.226792 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ecee9ae-a839-4edc-b06b-cc014cb7e7ff-kube-api-access-g7xqf\" (UniqueName: \"kubernetes.io/projected/5ecee9ae-a839-4edc-b06b-cc014cb7e7ff-kube-api-access-g7xqf\") pod \"5ecee9ae-a839-4edc-b06b-cc014cb7e7ff\" (UID: \"5ecee9ae-a839-4edc-b06b-cc014cb7e7ff\") " Oct 30 20:06:34.227009 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.226882 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ecee9ae-a839-4edc-b06b-cc014cb7e7ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecee9ae-a839-4edc-b06b-cc014cb7e7ff-plugins-config-volume\") pod \"5ecee9ae-a839-4edc-b06b-cc014cb7e7ff\" (UID: \"5ecee9ae-a839-4edc-b06b-cc014cb7e7ff\") " Oct 30 20:06:34.227157 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.227145 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f54b26b9-651b-4ba4-82a4-57819b4f02b4-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:34.227195 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.227160 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjnhb\" (UniqueName: \"kubernetes.io/projected/f54b26b9-651b-4ba4-82a4-57819b4f02b4-kube-api-access-tjnhb\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:34.227283 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.227262 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ecee9ae-a839-4edc-b06b-cc014cb7e7ff-plugins-config-volume" pod "5ecee9ae-a839-4edc-b06b-cc014cb7e7ff" (UID: "5ecee9ae-a839-4edc-b06b-cc014cb7e7ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:34.228961 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.228943 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecee9ae-a839-4edc-b06b-cc014cb7e7ff-kube-api-access-g7xqf" pod "5ecee9ae-a839-4edc-b06b-cc014cb7e7ff" (UID: "5ecee9ae-a839-4edc-b06b-cc014cb7e7ff"). InnerVolumeSpecName "kube-api-access-g7xqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:34.328165 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.328137 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecee9ae-a839-4edc-b06b-cc014cb7e7ff-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:34.328165 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.328163 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7xqf\" (UniqueName: \"kubernetes.io/projected/5ecee9ae-a839-4edc-b06b-cc014cb7e7ff-kube-api-access-g7xqf\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:34.680096 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.680054 4433 generic.go:356] "Generic (PLEG): container finished" podID="5ecee9ae-a839-4edc-b06b-cc014cb7e7ff" containerID="e328f93251d91e0deb76a6735b7f0e00b9dc38ab1732c2a819946ad63622431f" exitCode=0 Oct 30 20:06:34.680277 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.680112 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" Oct 30 20:06:34.680277 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.680139 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" event={"ID":"5ecee9ae-a839-4edc-b06b-cc014cb7e7ff","Type":"ContainerDied","Data":"e328f93251d91e0deb76a6735b7f0e00b9dc38ab1732c2a819946ad63622431f"} Oct 30 20:06:34.680277 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.680222 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t" event={"ID":"5ecee9ae-a839-4edc-b06b-cc014cb7e7ff","Type":"ContainerDied","Data":"58f05fc19823e83257cf23e8664a5fff57fefdde0af8ab634f95cfb32027d89d"} Oct 30 20:06:34.680277 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.680254 4433 scope.go:122] "RemoveContainer" containerID="e328f93251d91e0deb76a6735b7f0e00b9dc38ab1732c2a819946ad63622431f" Oct 30 20:06:34.681492 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.681433 4433 generic.go:356] "Generic (PLEG): container finished" podID="f54b26b9-651b-4ba4-82a4-57819b4f02b4" containerID="aaddbbf3c4840d1108cad0187494b149c59a80b614ee09146440a08d8c5dc9d2" exitCode=0 Oct 30 20:06:34.681492 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.681465 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" event={"ID":"f54b26b9-651b-4ba4-82a4-57819b4f02b4","Type":"ContainerDied","Data":"aaddbbf3c4840d1108cad0187494b149c59a80b614ee09146440a08d8c5dc9d2"} Oct 30 20:06:34.681492 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.681490 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" Oct 30 20:06:34.681653 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.681518 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd" event={"ID":"f54b26b9-651b-4ba4-82a4-57819b4f02b4","Type":"ContainerDied","Data":"ace17fb7a6c934f08695fda608d47e7ebc75392473d7dbfc237f4c935c37eb14"} Oct 30 20:06:34.689719 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.689696 4433 scope.go:122] "RemoveContainer" containerID="e328f93251d91e0deb76a6735b7f0e00b9dc38ab1732c2a819946ad63622431f" Oct 30 20:06:34.690039 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:06:34.690013 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e328f93251d91e0deb76a6735b7f0e00b9dc38ab1732c2a819946ad63622431f\": container with ID starting with e328f93251d91e0deb76a6735b7f0e00b9dc38ab1732c2a819946ad63622431f not found: ID does not exist" containerID="e328f93251d91e0deb76a6735b7f0e00b9dc38ab1732c2a819946ad63622431f" Oct 30 20:06:34.690104 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.690056 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e328f93251d91e0deb76a6735b7f0e00b9dc38ab1732c2a819946ad63622431f"} err="failed to get container status \"e328f93251d91e0deb76a6735b7f0e00b9dc38ab1732c2a819946ad63622431f\": rpc error: code = NotFound desc = could not find container \"e328f93251d91e0deb76a6735b7f0e00b9dc38ab1732c2a819946ad63622431f\": container with ID starting with e328f93251d91e0deb76a6735b7f0e00b9dc38ab1732c2a819946ad63622431f not found: ID does not exist" Oct 30 20:06:34.690104 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.690081 4433 scope.go:122] "RemoveContainer" containerID="aaddbbf3c4840d1108cad0187494b149c59a80b614ee09146440a08d8c5dc9d2" Oct 30 20:06:34.701836 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.701815 4433 scope.go:122] "RemoveContainer" containerID="aaddbbf3c4840d1108cad0187494b149c59a80b614ee09146440a08d8c5dc9d2" Oct 30 20:06:34.702164 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:06:34.702142 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaddbbf3c4840d1108cad0187494b149c59a80b614ee09146440a08d8c5dc9d2\": container with ID starting with aaddbbf3c4840d1108cad0187494b149c59a80b614ee09146440a08d8c5dc9d2 not found: ID does not exist" containerID="aaddbbf3c4840d1108cad0187494b149c59a80b614ee09146440a08d8c5dc9d2" Oct 30 20:06:34.702228 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.702174 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaddbbf3c4840d1108cad0187494b149c59a80b614ee09146440a08d8c5dc9d2"} err="failed to get container status \"aaddbbf3c4840d1108cad0187494b149c59a80b614ee09146440a08d8c5dc9d2\": rpc error: code = NotFound desc = could not find container \"aaddbbf3c4840d1108cad0187494b149c59a80b614ee09146440a08d8c5dc9d2\": container with ID starting with aaddbbf3c4840d1108cad0187494b149c59a80b614ee09146440a08d8c5dc9d2 not found: ID does not exist" Oct 30 20:06:34.846852 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.846818 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd"] Oct 30 20:06:34.895862 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.895826 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b5178f-7494d565d4-cqvqd"] Oct 30 20:06:34.957480 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.957400 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t"] Oct 30 20:06:34.963566 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:34.963532 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b5178f-9b899b9d7-jh66t"] Oct 30 20:06:35.213570 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:35.213487 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ecee9ae-a839-4edc-b06b-cc014cb7e7ff" path="/var/lib/kubelet/pods/5ecee9ae-a839-4edc-b06b-cc014cb7e7ff/volumes" Oct 30 20:06:35.214054 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:35.214036 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f54b26b9-651b-4ba4-82a4-57819b4f02b4" path="/var/lib/kubelet/pods/f54b26b9-651b-4ba4-82a4-57819b4f02b4/volumes" Oct 30 20:06:36.644599 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:36.644567 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:36.939581 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:36.939497 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44"] Oct 30 20:06:36.939748 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:36.939700 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" podUID="fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" containerName="istio-proxy" containerID="cri-o://81be2bc6c4e92f9a3900f58e6ec5162b23f65fc76b3c60bb22b1982248aefb6a" gracePeriod=30 Oct 30 20:06:36.965078 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:36.965026 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:37.181231 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:37.181198 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:42.206679 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.206615 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 20:06:42.307796 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.307758 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-data\") pod \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " Oct 30 20:06:42.307796 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.307798 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-credential-socket\") pod \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " Oct 30 20:06:42.308041 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.307831 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istiod-ca-cert\") pod \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " Oct 30 20:06:42.308041 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.307918 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-envoy\") pod \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " Oct 30 20:06:42.308041 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.307951 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-token\" (UniqueName: \"kubernetes.io/projected/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-token\") pod \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " Oct 30 20:06:42.308041 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.307996 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-workload-certs\") pod \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " Oct 30 20:06:42.308041 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.308028 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-podinfo\") pod \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " Oct 30 20:06:42.308301 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.308081 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-kube-api-access-7srjm\" (UniqueName: \"kubernetes.io/projected/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-kube-api-access-7srjm\") pod \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " Oct 30 20:06:42.308301 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.308096 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-data" pod "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" (UID: "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:42.308301 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.308115 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-credential-socket" pod "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" (UID: "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:42.308301 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.308158 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-workload-socket\") pod \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\" (UID: \"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6\") " Oct 30 20:06:42.308523 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.308328 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-workload-certs" pod "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" (UID: "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:42.308523 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.308352 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istiod-ca-cert" pod "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" (UID: "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:42.308788 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.308763 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-workload-socket" pod "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" (UID: "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:42.309605 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.309582 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:42.309705 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.309609 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:42.309705 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.309624 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:42.309705 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.309638 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:42.310447 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.310422 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-token" pod "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" (UID: "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:42.310536 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.310427 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-envoy" pod "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" (UID: "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:42.310536 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.310474 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-kube-api-access-7srjm" pod "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" (UID: "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6"). InnerVolumeSpecName "kube-api-access-7srjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:42.310702 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.310689 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-podinfo" pod "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" (UID: "fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:06:42.411125 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.411089 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:42.411125 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.411120 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:42.411319 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.411132 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7srjm\" (UniqueName: \"kubernetes.io/projected/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-kube-api-access-7srjm\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:42.411319 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.411142 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:42.411319 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.411151 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:06:42.721821 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.721788 4433 generic.go:356] "Generic (PLEG): container finished" podID="fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" containerID="81be2bc6c4e92f9a3900f58e6ec5162b23f65fc76b3c60bb22b1982248aefb6a" exitCode=0 Oct 30 20:06:42.722028 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.721834 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" Oct 30 20:06:42.722028 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.721870 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" event={"ID":"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6","Type":"ContainerDied","Data":"81be2bc6c4e92f9a3900f58e6ec5162b23f65fc76b3c60bb22b1982248aefb6a"} Oct 30 20:06:42.722028 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.721922 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44" event={"ID":"fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6","Type":"ContainerDied","Data":"07c1a069b9b99e32dfc2850733a1786aeba5a64a7e1aa9e74a2a96ab08e643fd"} Oct 30 20:06:42.722028 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.721943 4433 scope.go:122] "RemoveContainer" containerID="81be2bc6c4e92f9a3900f58e6ec5162b23f65fc76b3c60bb22b1982248aefb6a" Oct 30 20:06:42.731432 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.731411 4433 scope.go:122] "RemoveContainer" containerID="81be2bc6c4e92f9a3900f58e6ec5162b23f65fc76b3c60bb22b1982248aefb6a" Oct 30 20:06:42.731715 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:06:42.731696 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81be2bc6c4e92f9a3900f58e6ec5162b23f65fc76b3c60bb22b1982248aefb6a\": container with ID starting with 81be2bc6c4e92f9a3900f58e6ec5162b23f65fc76b3c60bb22b1982248aefb6a not found: ID does not exist" containerID="81be2bc6c4e92f9a3900f58e6ec5162b23f65fc76b3c60bb22b1982248aefb6a" Oct 30 20:06:42.731774 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.731723 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81be2bc6c4e92f9a3900f58e6ec5162b23f65fc76b3c60bb22b1982248aefb6a"} err="failed to get container status \"81be2bc6c4e92f9a3900f58e6ec5162b23f65fc76b3c60bb22b1982248aefb6a\": rpc error: code = NotFound desc = could not find container \"81be2bc6c4e92f9a3900f58e6ec5162b23f65fc76b3c60bb22b1982248aefb6a\": container with ID starting with 81be2bc6c4e92f9a3900f58e6ec5162b23f65fc76b3c60bb22b1982248aefb6a not found: ID does not exist" Oct 30 20:06:42.884873 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.884839 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44"] Oct 30 20:06:42.887884 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:42.887859 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-n7t44"] Oct 30 20:06:43.213063 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:43.213030 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" path="/var/lib/kubelet/pods/fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6/volumes" Oct 30 20:06:47.594347 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:47.594314 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:47.843949 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:47.843895 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:48.662434 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:48.662383 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:53.365701 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:06:53.365664 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:06:54.356906 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:54.356873 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:56.676761 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:06:56.676718 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:02.792804 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:02.792770 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:06.652012 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:06.651960 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:11.224241 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224206 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z"] Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224569 4433 cpu_manager.go:414] "containerMap: removing container" podUID="ff913782-d7e9-4110-a499-623fda834acd" containerName="istio-proxy" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224580 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff913782-d7e9-4110-a499-623fda834acd" containerName="istio-proxy" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224590 4433 cpu_manager.go:414] "containerMap: removing container" podUID="ded326eb-7db6-432f-8427-68833723a5fa" containerName="cleanup" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224596 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded326eb-7db6-432f-8427-68833723a5fa" containerName="cleanup" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224603 4433 cpu_manager.go:414] "containerMap: removing container" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224608 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224626 4433 cpu_manager.go:414] "containerMap: removing container" podUID="c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904" containerName="payload-processing" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224632 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904" containerName="payload-processing" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224642 4433 cpu_manager.go:414] "containerMap: removing container" podUID="0d290229-7e04-47a2-8c0a-c18988cd07f0" containerName="main" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224647 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d290229-7e04-47a2-8c0a-c18988cd07f0" containerName="main" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224654 4433 cpu_manager.go:414] "containerMap: removing container" podUID="efa7df32-302f-44e6-aef2-c4aff86c2b78" containerName="istio-proxy" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224659 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa7df32-302f-44e6-aef2-c4aff86c2b78" containerName="istio-proxy" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224670 4433 cpu_manager.go:414] "containerMap: removing container" podUID="edac81ed-b1b2-4ddb-a2ae-bfe897991191" containerName="payload-pre-processing" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224674 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edac81ed-b1b2-4ddb-a2ae-bfe897991191" containerName="payload-pre-processing" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224682 4433 cpu_manager.go:414] "containerMap: removing container" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224686 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224693 4433 cpu_manager.go:414] "containerMap: removing container" podUID="d9390ac4-cd34-46fa-ae2d-491bd2668c99" containerName="revoke-keys" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224698 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9390ac4-cd34-46fa-ae2d-491bd2668c99" containerName="revoke-keys" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224707 4433 cpu_manager.go:414] "containerMap: removing container" podUID="2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" containerName="cleanup" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224711 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" containerName="cleanup" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224718 4433 cpu_manager.go:414] "containerMap: removing container" podUID="5ecee9ae-a839-4edc-b06b-cc014cb7e7ff" containerName="payload-pre-processing" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224722 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ecee9ae-a839-4edc-b06b-cc014cb7e7ff" containerName="payload-pre-processing" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224729 4433 cpu_manager.go:414] "containerMap: removing container" podUID="c2ffbe2b-c076-4ae7-994b-d8ab2db02185" containerName="main" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224733 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ffbe2b-c076-4ae7-994b-d8ab2db02185" containerName="main" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224741 4433 cpu_manager.go:414] "containerMap: removing container" podUID="f54b26b9-651b-4ba4-82a4-57819b4f02b4" containerName="payload-processing" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224745 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54b26b9-651b-4ba4-82a4-57819b4f02b4" containerName="payload-processing" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224754 4433 cpu_manager.go:414] "containerMap: removing container" podUID="f1e0904a-f62b-4828-b130-fdef755b3831" containerName="istio-proxy" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224758 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e0904a-f62b-4828-b130-fdef755b3831" containerName="istio-proxy" Oct 30 20:07:11.224750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224766 4433 cpu_manager.go:414] "containerMap: removing container" podUID="23196d8b-4348-4e50-9f54-5c44c4b6b35c" containerName="payload-processing" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224770 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23196d8b-4348-4e50-9f54-5c44c4b6b35c" containerName="payload-processing" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224778 4433 cpu_manager.go:414] "containerMap: removing container" podUID="fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" containerName="istio-proxy" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224782 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" containerName="istio-proxy" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224788 4433 cpu_manager.go:414] "containerMap: removing container" podUID="1bd1c66b-4d08-4aca-ab6b-05b7267e7821" containerName="main" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224792 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd1c66b-4d08-4aca-ab6b-05b7267e7821" containerName="main" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224806 4433 cpu_manager.go:414] "containerMap: removing container" podUID="75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" containerName="istio-proxy" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224810 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" containerName="istio-proxy" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224818 4433 cpu_manager.go:414] "containerMap: removing container" podUID="bc78983a-aa83-48e3-b0c3-95148da6dfc1" containerName="discovery" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224822 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc78983a-aa83-48e3-b0c3-95148da6dfc1" containerName="discovery" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224829 4433 cpu_manager.go:414] "containerMap: removing container" podUID="6c890add-9063-4067-bb5e-65324cf99b3d" containerName="main" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224832 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c890add-9063-4067-bb5e-65324cf99b3d" containerName="main" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224884 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="23196d8b-4348-4e50-9f54-5c44c4b6b35c" containerName="payload-processing" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224894 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdd5ed0-278d-48ba-b7bc-b6bd929e23c6" containerName="istio-proxy" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224900 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224905 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd1c66b-4d08-4aca-ab6b-05b7267e7821" containerName="main" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224912 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c1ea2e-e33e-4828-abd3-266fcc1a8dbb" containerName="istio-proxy" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224918 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc78983a-aa83-48e3-b0c3-95148da6dfc1" containerName="discovery" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224924 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c890add-9063-4067-bb5e-65324cf99b3d" containerName="main" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224929 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb12b8c-9c22-4c09-8a9a-7637e3ce74ac" containerName="cleanup" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224934 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f4e9af-3d6f-499c-a4b8-a5e8c47fd904" containerName="payload-processing" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224939 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d290229-7e04-47a2-8c0a-c18988cd07f0" containerName="main" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224944 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="edac81ed-b1b2-4ddb-a2ae-bfe897991191" containerName="payload-pre-processing" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224949 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224954 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224959 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3af9891c-2337-4531-a662-1eb43724b440" containerName="istio-proxy" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224964 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9390ac4-cd34-46fa-ae2d-491bd2668c99" containerName="revoke-keys" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224969 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ecee9ae-a839-4edc-b06b-cc014cb7e7ff" containerName="payload-pre-processing" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224973 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2ffbe2b-c076-4ae7-994b-d8ab2db02185" containerName="main" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224992 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54b26b9-651b-4ba4-82a4-57819b4f02b4" containerName="payload-processing" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.224997 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff913782-d7e9-4110-a499-623fda834acd" containerName="istio-proxy" Oct 30 20:07:11.225729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.225003 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded326eb-7db6-432f-8427-68833723a5fa" containerName="cleanup" Oct 30 20:07:11.228063 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.228047 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.268001 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.267955 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.268137 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.268060 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.268137 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.268115 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd6qn\" (UniqueName: \"kubernetes.io/projected/485179f9-ab6f-4ff6-8520-437297c69df5-kube-api-access-bd6qn\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.268217 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.268137 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.268217 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.268156 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.268217 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.268181 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/485179f9-ab6f-4ff6-8520-437297c69df5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.268334 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.268230 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/485179f9-ab6f-4ff6-8520-437297c69df5-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.268379 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.268345 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.268379 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.268371 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/485179f9-ab6f-4ff6-8520-437297c69df5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.273464 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.273437 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt"] Oct 30 20:07:11.273829 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.273813 4433 cpu_manager.go:414] "containerMap: removing container" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" Oct 30 20:07:11.273893 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.273832 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" Oct 30 20:07:11.273893 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.273850 4433 cpu_manager.go:414] "containerMap: removing container" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" Oct 30 20:07:11.273893 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.273858 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" Oct 30 20:07:11.274039 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.273951 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0e2f76e-309b-46e6-8ee8-478c0def15fc" containerName="maas-api" Oct 30 20:07:11.278254 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.278224 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.331916 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.331890 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z"] Oct 30 20:07:11.338526 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.338495 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt"] Oct 30 20:07:11.369100 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369074 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd6qn\" (UniqueName: \"kubernetes.io/projected/485179f9-ab6f-4ff6-8520-437297c69df5-kube-api-access-bd6qn\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.369226 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369109 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.369226 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369129 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.369226 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369150 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.369226 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369216 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.369398 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369245 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/485179f9-ab6f-4ff6-8520-437297c69df5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.369398 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369286 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/485179f9-ab6f-4ff6-8520-437297c69df5-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.369398 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369313 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.369398 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369369 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.369603 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369449 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.369603 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369507 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/485179f9-ab6f-4ff6-8520-437297c69df5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.369603 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369548 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdd6z\" (UniqueName: \"kubernetes.io/projected/3485be44-cb71-42c4-ab8e-dfcd161784cd-kube-api-access-bdd6z\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.369731 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369601 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.369731 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369634 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.369731 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369688 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.369837 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369732 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.369837 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369814 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.369916 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369865 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.369916 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.369896 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3485be44-cb71-42c4-ab8e-dfcd161784cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.370026 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.370007 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/485179f9-ab6f-4ff6-8520-437297c69df5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.370136 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.370113 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.370201 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.370125 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.370351 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.370331 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.372010 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.371991 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.372058 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.371996 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/485179f9-ab6f-4ff6-8520-437297c69df5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.420087 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.420054 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:11.425280 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.425250 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/485179f9-ab6f-4ff6-8520-437297c69df5-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.425415 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.425396 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd6qn\" (UniqueName: \"kubernetes.io/projected/485179f9-ab6f-4ff6-8520-437297c69df5-kube-api-access-bd6qn\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ld68z\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.470448 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.470407 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.470448 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.470450 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3485be44-cb71-42c4-ab8e-dfcd161784cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.470719 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.470497 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.470719 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.470526 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.470719 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.470559 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.470719 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.470628 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.470719 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.470673 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdd6z\" (UniqueName: \"kubernetes.io/projected/3485be44-cb71-42c4-ab8e-dfcd161784cd-kube-api-access-bdd6z\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.470975 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.470728 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.470975 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.470759 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.470975 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.470873 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.470975 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.470952 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.471212 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.471100 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.471252 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.471208 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.471286 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.471271 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3485be44-cb71-42c4-ab8e-dfcd161784cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.473150 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.473128 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.473253 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.473207 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.527047 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.526949 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.527186 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.527084 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdd6z\" (UniqueName: \"kubernetes.io/projected/3485be44-cb71-42c4-ab8e-dfcd161784cd-kube-api-access-bdd6z\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fzmkt\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.540906 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.540860 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:11.592447 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.592413 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:11.962871 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.962812 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z"] Oct 30 20:07:11.965478 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:07:11.965446 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485179f9_ab6f_4ff6_8520_437297c69df5.slice/crio-52cf0c5637d37eb75cda2ece668c56565b78cf4da603e58c0b7ae1a504b69fa4 WatchSource:0}: Error finding container 52cf0c5637d37eb75cda2ece668c56565b78cf4da603e58c0b7ae1a504b69fa4: Status 404 returned error can't find the container with id 52cf0c5637d37eb75cda2ece668c56565b78cf4da603e58c0b7ae1a504b69fa4 Oct 30 20:07:11.967544 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.967514 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:11.967622 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.967587 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:11.967622 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:11.967614 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:12.068607 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:12.068580 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt"] Oct 30 20:07:12.070816 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:07:12.070785 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3485be44_cb71_42c4_ab8e_dfcd161784cd.slice/crio-dfe105ffc3306c0975298fe47cc1e6230a7f494feedb00c3c8d0a59621cc25f7 WatchSource:0}: Error finding container dfe105ffc3306c0975298fe47cc1e6230a7f494feedb00c3c8d0a59621cc25f7: Status 404 returned error can't find the container with id dfe105ffc3306c0975298fe47cc1e6230a7f494feedb00c3c8d0a59621cc25f7 Oct 30 20:07:12.073227 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:12.073193 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:12.073323 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:12.073268 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:12.073323 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:12.073298 4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:07:12.865791 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:12.865750 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" event={"ID":"485179f9-ab6f-4ff6-8520-437297c69df5","Type":"ContainerStarted","Data":"9728b3783d1c2fa985397ea1d3f8964ed719d6967ce9a0bb0b724cee6c4758fe"} Oct 30 20:07:12.866299 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:12.865815 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" event={"ID":"485179f9-ab6f-4ff6-8520-437297c69df5","Type":"ContainerStarted","Data":"52cf0c5637d37eb75cda2ece668c56565b78cf4da603e58c0b7ae1a504b69fa4"} Oct 30 20:07:12.867401 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:12.867372 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" event={"ID":"3485be44-cb71-42c4-ab8e-dfcd161784cd","Type":"ContainerStarted","Data":"680f2661ae4ffb440da502aab71c31417f02d5db2f57a1ccdea6941d32a663ae"} Oct 30 20:07:12.867487 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:12.867419 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" event={"ID":"3485be44-cb71-42c4-ab8e-dfcd161784cd","Type":"ContainerStarted","Data":"dfe105ffc3306c0975298fe47cc1e6230a7f494feedb00c3c8d0a59621cc25f7"} Oct 30 20:07:13.039409 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:13.039339 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" podStartSLOduration=2.039325273 podStartE2EDuration="2.039325273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:13.034123967 +0000 UTC m=+3079.968911483" watchObservedRunningTime="2026-10-30 20:07:13.039325273 +0000 UTC m=+3079.974112796" Oct 30 20:07:13.169545 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:13.169440 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" podStartSLOduration=2.169023961 podStartE2EDuration="2.169023961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:13.167105573 +0000 UTC m=+3080.101893097" watchObservedRunningTime="2026-10-30 20:07:13.169023961 +0000 UTC m=+3080.103811555" Oct 30 20:07:13.541384 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:13.541274 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:13.543104 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:13.543074 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ld68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:07:13.543243 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:13.543127 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" podUID="485179f9-ab6f-4ff6-8520-437297c69df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 20:07:13.610002 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:13.606045 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:13.610002 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:13.606204 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fzmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 20:07:13.610002 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:13.606255 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" podUID="3485be44-cb71-42c4-ab8e-dfcd161784cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 20:07:14.541665 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:14.541502 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ld68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:07:14.541665 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:14.541561 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" podUID="485179f9-ab6f-4ff6-8520-437297c69df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 20:07:14.600004 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:14.596469 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fzmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 20:07:14.600004 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:14.596544 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" podUID="3485be44-cb71-42c4-ab8e-dfcd161784cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 20:07:15.541805 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:15.541767 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ld68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:07:15.542239 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:15.541843 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" podUID="485179f9-ab6f-4ff6-8520-437297c69df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 20:07:15.602012 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:15.601394 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fzmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 20:07:15.602012 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:15.601473 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" podUID="3485be44-cb71-42c4-ab8e-dfcd161784cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 20:07:16.543036 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:16.541607 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ld68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:07:16.543036 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:16.541680 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" podUID="485179f9-ab6f-4ff6-8520-437297c69df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 20:07:16.597873 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:16.594398 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fzmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 20:07:16.597873 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:16.594465 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" podUID="3485be44-cb71-42c4-ab8e-dfcd161784cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 20:07:17.544000 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:17.543485 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ld68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:07:17.544000 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:17.543565 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" podUID="485179f9-ab6f-4ff6-8520-437297c69df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 20:07:17.593513 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:17.593474 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fzmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 20:07:17.597026 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:17.593771 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" podUID="3485be44-cb71-42c4-ab8e-dfcd161784cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 20:07:18.541761 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:18.541716 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ld68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:07:18.542001 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:18.541785 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" podUID="485179f9-ab6f-4ff6-8520-437297c69df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 20:07:18.593372 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:18.593322 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fzmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 20:07:18.593885 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:18.593393 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" podUID="3485be44-cb71-42c4-ab8e-dfcd161784cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 20:07:20.542537 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:20.542501 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ld68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:20.543000 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:20.542586 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" podUID="485179f9-ab6f-4ff6-8520-437297c69df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:20.593627 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:20.593585 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fzmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:20.593806 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:20.593679 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" podUID="3485be44-cb71-42c4-ab8e-dfcd161784cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:21.584150 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:21.584107 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ld68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:21.584801 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:21.584747 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" podUID="485179f9-ab6f-4ff6-8520-437297c69df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:21.635162 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:21.635122 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fzmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:21.635456 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:21.635426 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" podUID="3485be44-cb71-42c4-ab8e-dfcd161784cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:22.320345 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:22.320311 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:22.320904 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:22.320882 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:22.321647 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:22.321626 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:07:22.636767 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:22.636678 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fzmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:22.636767 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:22.636746 4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" podUID="3485be44-cb71-42c4-ab8e-dfcd161784cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:22.687740 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:22.687702 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:22.688226 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:22.688199 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:22.688860 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:22.688844 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:07:23.365502 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:07:23.365461 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:07:25.052456 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:25.052426 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb"] Oct 30 20:07:25.101619 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:25.101590 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddb5f24e-66ghb"] Oct 30 20:07:25.211907 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:25.211867 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9390ac4-cd34-46fa-ae2d-491bd2668c99" path="/var/lib/kubelet/pods/d9390ac4-cd34-46fa-ae2d-491bd2668c99/volumes" Oct 30 20:07:27.737838 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:27.737803 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:29.569323 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:29.569290 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:33.357568 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:33.357519 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:53.366622 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:07:53.366582 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:07:57.014776 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:07:57.014738 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:03.136750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:08:03.136706 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:23.367391 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:08:23.367349 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:08:31.561896 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:08:31.561853 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:42.709457 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:08:42.709416 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:53.368272 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:08:53.368223 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:08:54.041078 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:08:54.041043 4433 scope.go:122] "RemoveContainer" containerID="73b466979f0d630e737fef99653e68fe06425bcebd90e10d7be05fa84dfa2c48" Oct 30 20:08:57.362764 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:08:57.362730 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:01.708273 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:09:01.708242 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:03.472998 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:09:03.472951 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:07.308378 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:09:07.308295 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:23.367501 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:09:23.367461 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:09:26.857325 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:09:26.857292 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:38.706945 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:09:38.706888 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:53.367231 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:09:53.367187 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ld68z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:53.367656 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:09:53.367250 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" podUID="485179f9-ab6f-4ff6-8520-437297c69df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:53.373433 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:09:53.373397 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:10:03.814053 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:10:03.814001 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:06.079415 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:10:06.079377 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:06.306861 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:10:06.306821 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:17.695447 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:10:17.695382 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:23.368645 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:10:23.368600 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:10:39.050615 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:10:39.050521 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:53.366196 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:10:53.366151 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:11:04.181507 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:11:04.181466 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:14.796844 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:11:14.796771 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:23.366552 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:11:23.366510 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:11:49.439464 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:11:49.439423 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:53.369749 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:11:53.369706 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:12:04.495784 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:12:04.495744 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:23.038820 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:12:23.038785 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:23.366379 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:12:23.366344 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:12:49.723030 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:12:49.722960 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:53.366251 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:12:53.366212 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:12:58.390361 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:12:58.390322 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:59.207375 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:12:59.207340 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:14.831925 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:13:14.831889 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:23.366752 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:13:23.366715 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:13:27.510243 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:13:27.510208 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:33.375724 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:13:33.375688 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:50.065526 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:13:50.065420 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:53.367294 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:13:53.367254 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:13:59.138571 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:13:59.138537 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:59.929327 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:13:59.929291 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:15.171632 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:14:15.171597 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:23.366911 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:14:23.366873 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:14:24.939880 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:14:24.939838 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:25.736653 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:14:25.736609 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:33.714029 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:14:33.713967 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:50.396526 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:14:50.396488 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:53.366950 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:14:53.366915 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:14:59.862718 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:14:59.862681 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:00.471204 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:00.471157 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:00.956055 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:00.956021 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:04.677436 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:04.677396 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:04.904466 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:04.904371 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:08.055213 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.055179 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:08.287627 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.287591 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:08.377337 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.377302 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd"] Oct 30 20:15:08.380948 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.380929 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" Oct 30 20:15:08.434927 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.434885 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec34b91-6b91-4031-ac85-eeb311bf4b0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd\" (UID: \"6ec34b91-6b91-4031-ac85-eeb311bf4b0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" Oct 30 20:15:08.435141 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.434944 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q848d\" (UniqueName: \"kubernetes.io/projected/6ec34b91-6b91-4031-ac85-eeb311bf4b0f-kube-api-access-q848d\") pod \"payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd\" (UID: \"6ec34b91-6b91-4031-ac85-eeb311bf4b0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" Oct 30 20:15:08.435634 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.435603 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-f1aa42\"" Oct 30 20:15:08.487159 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.487126 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd"] Oct 30 20:15:08.531682 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.531649 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258"] Oct 30 20:15:08.542573 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.542370 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec34b91-6b91-4031-ac85-eeb311bf4b0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd\" (UID: \"6ec34b91-6b91-4031-ac85-eeb311bf4b0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" Oct 30 20:15:08.542772 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.542623 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q848d\" (UniqueName: \"kubernetes.io/projected/6ec34b91-6b91-4031-ac85-eeb311bf4b0f-kube-api-access-q848d\") pod \"payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd\" (UID: \"6ec34b91-6b91-4031-ac85-eeb311bf4b0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" Oct 30 20:15:08.543792 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.543766 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec34b91-6b91-4031-ac85-eeb311bf4b0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd\" (UID: \"6ec34b91-6b91-4031-ac85-eeb311bf4b0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" Oct 30 20:15:08.544091 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.544071 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" Oct 30 20:15:08.593140 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.593103 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258"] Oct 30 20:15:08.595742 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.595712 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-59fc1e74-dockercfg-ljptf\"" Oct 30 20:15:08.599143 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.599116 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q848d\" (UniqueName: \"kubernetes.io/projected/6ec34b91-6b91-4031-ac85-eeb311bf4b0f-kube-api-access-q848d\") pod \"payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd\" (UID: \"6ec34b91-6b91-4031-ac85-eeb311bf4b0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" Oct 30 20:15:08.694255 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.694163 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" Oct 30 20:15:08.744691 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.744651 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t476\" (UniqueName: \"kubernetes.io/projected/9f02605e-49f9-4bee-908e-d914824af9f4-kube-api-access-5t476\") pod \"payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258\" (UID: \"9f02605e-49f9-4bee-908e-d914824af9f4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" Oct 30 20:15:08.744691 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.744730 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f02605e-49f9-4bee-908e-d914824af9f4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258\" (UID: \"9f02605e-49f9-4bee-908e-d914824af9f4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" Oct 30 20:15:08.845677 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.845639 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t476\" (UniqueName: \"kubernetes.io/projected/9f02605e-49f9-4bee-908e-d914824af9f4-kube-api-access-5t476\") pod \"payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258\" (UID: \"9f02605e-49f9-4bee-908e-d914824af9f4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" Oct 30 20:15:08.845890 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.845704 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f02605e-49f9-4bee-908e-d914824af9f4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258\" (UID: \"9f02605e-49f9-4bee-908e-d914824af9f4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" Oct 30 20:15:08.846467 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.846444 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f02605e-49f9-4bee-908e-d914824af9f4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258\" (UID: \"9f02605e-49f9-4bee-908e-d914824af9f4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" Oct 30 20:15:08.903379 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:08.903345 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t476\" (UniqueName: \"kubernetes.io/projected/9f02605e-49f9-4bee-908e-d914824af9f4-kube-api-access-5t476\") pod \"payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258\" (UID: \"9f02605e-49f9-4bee-908e-d914824af9f4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" Oct 30 20:15:09.066688 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:09.066658 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd"] Oct 30 20:15:09.069395 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:15:09.069364 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec34b91_6b91_4031_ac85_eeb311bf4b0f.slice/crio-02500d67e759fb556ed9874968849667d5a4f3f577d0674aa2bd5c00033a5f44 WatchSource:0}: Error finding container 02500d67e759fb556ed9874968849667d5a4f3f577d0674aa2bd5c00033a5f44: Status 404 returned error can't find the container with id 02500d67e759fb556ed9874968849667d5a4f3f577d0674aa2bd5c00033a5f44 Oct 30 20:15:09.157771 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:09.157741 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" Oct 30 20:15:09.533688 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:09.533603 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258"] Oct 30 20:15:09.536806 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:15:09.536772 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f02605e_49f9_4bee_908e_d914824af9f4.slice/crio-7ef3e8eb63145c5a45cced21577eb15ad3c7629354ec6cf9792716bf65319ebf WatchSource:0}: Error finding container 7ef3e8eb63145c5a45cced21577eb15ad3c7629354ec6cf9792716bf65319ebf: Status 404 returned error can't find the container with id 7ef3e8eb63145c5a45cced21577eb15ad3c7629354ec6cf9792716bf65319ebf Oct 30 20:15:09.631894 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:09.631863 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" event={"ID":"9f02605e-49f9-4bee-908e-d914824af9f4","Type":"ContainerStarted","Data":"25eddddae36d3a1f4ff6677f8a0785acc7f5fe642b37bd9e24cdc0b152c2f9fb"} Oct 30 20:15:09.632088 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:09.631914 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" event={"ID":"9f02605e-49f9-4bee-908e-d914824af9f4","Type":"ContainerStarted","Data":"7ef3e8eb63145c5a45cced21577eb15ad3c7629354ec6cf9792716bf65319ebf"} Oct 30 20:15:09.632088 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:09.631947 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" Oct 30 20:15:09.633438 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:09.633413 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" event={"ID":"6ec34b91-6b91-4031-ac85-eeb311bf4b0f","Type":"ContainerStarted","Data":"da3cc52f4a6ca4a3b65716969a978c8a07b510864e83d0e6eb636f65142be896"} Oct 30 20:15:09.633540 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:09.633452 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" event={"ID":"6ec34b91-6b91-4031-ac85-eeb311bf4b0f","Type":"ContainerStarted","Data":"02500d67e759fb556ed9874968849667d5a4f3f577d0674aa2bd5c00033a5f44"} Oct 30 20:15:09.633540 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:09.633499 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" Oct 30 20:15:09.747382 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:09.747327 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" podStartSLOduration=1.74731283 podStartE2EDuration="1.74731283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:09.745626119 +0000 UTC m=+3556.680413644" watchObservedRunningTime="2026-10-30 20:15:09.74731283 +0000 UTC m=+3556.682100357" Oct 30 20:15:09.861195 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:09.861126 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" podStartSLOduration=1.8611047699999999 podStartE2EDuration="1.86110477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:09.85871038 +0000 UTC m=+3556.793497934" watchObservedRunningTime="2026-10-30 20:15:09.86110477 +0000 UTC m=+3556.795892307" Oct 30 20:15:10.568812 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:10.568777 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:10.787448 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:10.787412 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:15.582439 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:15.582403 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:15.808289 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:15.808233 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:20.296813 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:20.296771 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:20.642098 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:20.642064 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" Oct 30 20:15:20.642282 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:20.642132 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" Oct 30 20:15:23.366252 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:15:23.366213 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:15:24.923436 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:24.923360 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:29.617897 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:29.617860 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:29.889498 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:29.889407 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:30.242661 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:30.242574 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:30.474482 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:30.474412 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:34.447434 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:34.447400 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:39.063774 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:39.063715 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:43.259354 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.259311 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl"] Oct 30 20:15:43.263005 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.262971 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.309573 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.309532 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-home\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.309777 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.309652 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-model-cache\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.309864 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.309780 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8038f47-7938-41c4-9fa9-9ecf844ed599-tls-certs\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.309864 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.309826 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-tmp-dir\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.309993 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.309952 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j26m6\" (UniqueName: \"kubernetes.io/projected/f8038f47-7938-41c4-9fa9-9ecf844ed599-kube-api-access-j26m6\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.310052 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.310016 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-dshm\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.364248 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.364169 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f1aa42\"/\"kube-root-ca.crt\"" Oct 30 20:15:43.364435 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.364382 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f1aa42\"/\"openshift-service-ca.crt\"" Oct 30 20:15:43.364435 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.364408 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f1aa42\"/\"default-dockercfg-twv4n\"" Oct 30 20:15:43.364576 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.364502 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f1aa42\"/\"simulator-single-main-f1aa42-kserve-self-signed-certs\"" Oct 30 20:15:43.366622 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.366603 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl"] Oct 30 20:15:43.411876 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.411344 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8038f47-7938-41c4-9fa9-9ecf844ed599-tls-certs\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.411876 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.411431 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-tmp-dir\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.411876 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.411547 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j26m6\" (UniqueName: \"kubernetes.io/projected/f8038f47-7938-41c4-9fa9-9ecf844ed599-kube-api-access-j26m6\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.412302 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.411603 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-dshm\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.412452 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.412387 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-home\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.412523 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.412449 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-model-cache\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.413063 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.413035 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-model-cache\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.413330 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.413308 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-tmp-dir\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.413624 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.413591 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-home\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.416555 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.416530 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-dshm\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.420724 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.420672 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8038f47-7938-41c4-9fa9-9ecf844ed599-tls-certs\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.437545 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.437515 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:43.469933 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.469904 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j26m6\" (UniqueName: \"kubernetes.io/projected/f8038f47-7938-41c4-9fa9-9ecf844ed599-kube-api-access-j26m6\") pod \"simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.574926 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.574891 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:43.749352 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.749316 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:43.954712 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:43.954682 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl"] Oct 30 20:15:43.957039 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:15:43.957007 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8038f47_7938_41c4_9fa9_9ecf844ed599.slice/crio-1a74d4752769b3d5e8541918dcadf2c7e8ae4cfeafb82dd546815f6e26205072 WatchSource:0}: Error finding container 1a74d4752769b3d5e8541918dcadf2c7e8ae4cfeafb82dd546815f6e26205072: Status 404 returned error can't find the container with id 1a74d4752769b3d5e8541918dcadf2c7e8ae4cfeafb82dd546815f6e26205072 Oct 30 20:15:44.056180 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:44.056144 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:44.803920 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:44.803879 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" event={"ID":"f8038f47-7938-41c4-9fa9-9ecf844ed599","Type":"ContainerStarted","Data":"35d401dbde5632f8ce3b0f393365cd3250438df56e033dd2be77e2accdcf5a5f"} Oct 30 20:15:44.804345 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:44.803931 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" event={"ID":"f8038f47-7938-41c4-9fa9-9ecf844ed599","Type":"ContainerStarted","Data":"1a74d4752769b3d5e8541918dcadf2c7e8ae4cfeafb82dd546815f6e26205072"} Oct 30 20:15:44.921848 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:44.921788 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" podStartSLOduration=1.92177254 podStartE2EDuration="1.92177254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:44.920569165 +0000 UTC m=+3591.855356714" watchObservedRunningTime="2026-10-30 20:15:44.92177254 +0000 UTC m=+3591.856560064" Oct 30 20:15:45.575241 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:45.575204 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:48.392806 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:48.392768 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:52.971549 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:52.971511 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:53.366589 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:15:53.366553 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:15:53.575890 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:53.575850 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:53.588626 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:53.588588 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:53.866807 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:53.866775 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:15:57.598767 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:15:57.598723 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:02.158693 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:02.158656 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:06.945261 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:06.945222 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:11.420374 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:11.420339 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:15.837702 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:15.837667 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:16.231577 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:16.231490 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:20.799994 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:20.799954 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:23.366918 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:16:23.366869 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:16:25.448520 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:25.448488 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:30.346362 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:30.346310 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:34.938219 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:34.938120 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:35.153965 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:35.153921 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:39.785279 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:39.785246 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:39.959190 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:39.959152 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl"] Oct 30 20:16:40.116760 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:40.116729 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:44.397605 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:44.397571 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:44.629244 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:44.629208 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:47.008671 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:47.008635 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258"] Oct 30 20:16:47.118183 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:47.118152 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd"] Oct 30 20:16:47.463760 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:47.463720 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:47.682387 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:47.682352 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:49.083686 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:49.083650 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:52.009212 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:52.009158 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" podUID="9f02605e-49f9-4bee-908e-d914824af9f4" containerName="payload-processing" containerID="cri-o://25eddddae36d3a1f4ff6677f8a0785acc7f5fe642b37bd9e24cdc0b152c2f9fb" gracePeriod=25 Oct 30 20:16:52.119054 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:52.119014 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" podUID="6ec34b91-6b91-4031-ac85-eeb311bf4b0f" containerName="payload-pre-processing" containerID="cri-o://da3cc52f4a6ca4a3b65716969a978c8a07b510864e83d0e6eb636f65142be896" gracePeriod=25 Oct 30 20:16:53.219248 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:53.219205 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:53.367640 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:16:53.367598 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:16:54.673757 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:54.673718 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:54.943894 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:54.943800 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:54.979120 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:54.979067 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" podUID="f8038f47-7938-41c4-9fa9-9ecf844ed599" containerName="main" containerID="cri-o://35d401dbde5632f8ce3b0f393365cd3250438df56e033dd2be77e2accdcf5a5f" gracePeriod=45 Oct 30 20:16:55.168820 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.168787 4433 generic.go:356] "Generic (PLEG): container finished" podID="f8038f47-7938-41c4-9fa9-9ecf844ed599" containerID="35d401dbde5632f8ce3b0f393365cd3250438df56e033dd2be77e2accdcf5a5f" exitCode=0 Oct 30 20:16:55.168971 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.168861 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" event={"ID":"f8038f47-7938-41c4-9fa9-9ecf844ed599","Type":"ContainerDied","Data":"35d401dbde5632f8ce3b0f393365cd3250438df56e033dd2be77e2accdcf5a5f"} Oct 30 20:16:55.236872 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.236841 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:16:55.347124 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.347084 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-home\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-home\") pod \"f8038f47-7938-41c4-9fa9-9ecf844ed599\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " Oct 30 20:16:55.347124 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.347126 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-model-cache\") pod \"f8038f47-7938-41c4-9fa9-9ecf844ed599\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " Oct 30 20:16:55.347363 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.347186 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-tmp-dir\") pod \"f8038f47-7938-41c4-9fa9-9ecf844ed599\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " Oct 30 20:16:55.347363 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.347244 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-dshm\") pod \"f8038f47-7938-41c4-9fa9-9ecf844ed599\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " Oct 30 20:16:55.347363 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.347339 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8038f47-7938-41c4-9fa9-9ecf844ed599-kube-api-access-j26m6\" (UniqueName: \"kubernetes.io/projected/f8038f47-7938-41c4-9fa9-9ecf844ed599-kube-api-access-j26m6\") pod \"f8038f47-7938-41c4-9fa9-9ecf844ed599\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " Oct 30 20:16:55.347532 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.347412 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8038f47-7938-41c4-9fa9-9ecf844ed599-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8038f47-7938-41c4-9fa9-9ecf844ed599-tls-certs\") pod \"f8038f47-7938-41c4-9fa9-9ecf844ed599\" (UID: \"f8038f47-7938-41c4-9fa9-9ecf844ed599\") " Oct 30 20:16:55.347532 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.347436 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-tmp-dir" pod "f8038f47-7938-41c4-9fa9-9ecf844ed599" (UID: "f8038f47-7938-41c4-9fa9-9ecf844ed599"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:55.347532 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.347452 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-home" pod "f8038f47-7938-41c4-9fa9-9ecf844ed599" (UID: "f8038f47-7938-41c4-9fa9-9ecf844ed599"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:55.347532 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.347466 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-model-cache" pod "f8038f47-7938-41c4-9fa9-9ecf844ed599" (UID: "f8038f47-7938-41c4-9fa9-9ecf844ed599"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:55.347776 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.347755 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-home\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:16:55.347818 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.347781 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-model-cache\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:16:55.347818 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.347795 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-tmp-dir\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:16:55.349634 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.349604 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8038f47-7938-41c4-9fa9-9ecf844ed599-tls-certs" pod "f8038f47-7938-41c4-9fa9-9ecf844ed599" (UID: "f8038f47-7938-41c4-9fa9-9ecf844ed599"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:16:55.349758 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.349636 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-dshm" pod "f8038f47-7938-41c4-9fa9-9ecf844ed599" (UID: "f8038f47-7938-41c4-9fa9-9ecf844ed599"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:55.349758 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.349678 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8038f47-7938-41c4-9fa9-9ecf844ed599-kube-api-access-j26m6" pod "f8038f47-7938-41c4-9fa9-9ecf844ed599" (UID: "f8038f47-7938-41c4-9fa9-9ecf844ed599"). InnerVolumeSpecName "kube-api-access-j26m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:55.449034 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.448967 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8038f47-7938-41c4-9fa9-9ecf844ed599-tls-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:16:55.449034 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.449031 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8038f47-7938-41c4-9fa9-9ecf844ed599-dshm\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:16:55.449034 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:55.449041 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j26m6\" (UniqueName: \"kubernetes.io/projected/f8038f47-7938-41c4-9fa9-9ecf844ed599-kube-api-access-j26m6\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:16:56.032577 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:56.032537 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:56.175789 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:56.175758 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" Oct 30 20:16:56.175973 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:56.175792 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl" event={"ID":"f8038f47-7938-41c4-9fa9-9ecf844ed599","Type":"ContainerDied","Data":"1a74d4752769b3d5e8541918dcadf2c7e8ae4cfeafb82dd546815f6e26205072"} Oct 30 20:16:56.175973 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:56.175849 4433 scope.go:122] "RemoveContainer" containerID="35d401dbde5632f8ce3b0f393365cd3250438df56e033dd2be77e2accdcf5a5f" Oct 30 20:16:56.253324 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:56.253257 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:56.338522 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:56.338478 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl"] Oct 30 20:16:56.387670 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:56.387632 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-f1aa42/simulator-single-main-f1aa42-kserve-54bb88b895-8xvkl"] Oct 30 20:16:57.212588 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:16:57.212554 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8038f47-7938-41c4-9fa9-9ecf844ed599" path="/var/lib/kubelet/pods/f8038f47-7938-41c4-9fa9-9ecf844ed599/volumes" Oct 30 20:17:00.641243 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:00.641207 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" podUID="6ec34b91-6b91-4031-ac85-eeb311bf4b0f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:17:00.641665 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:00.641201 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" podUID="9f02605e-49f9-4bee-908e-d914824af9f4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:17:02.030575 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:02.030543 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:02.261852 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:02.261817 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:02.489541 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:02.489511 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:10.640812 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:10.640766 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" podUID="6ec34b91-6b91-4031-ac85-eeb311bf4b0f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:17:10.641346 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:10.640823 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" podUID="9f02605e-49f9-4bee-908e-d914824af9f4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:17:12.256758 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.256619 4433 generic.go:356] "Generic (PLEG): container finished" podID="9f02605e-49f9-4bee-908e-d914824af9f4" containerID="25eddddae36d3a1f4ff6677f8a0785acc7f5fe642b37bd9e24cdc0b152c2f9fb" exitCode=0 Oct 30 20:17:12.256758 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.256685 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" event={"ID":"9f02605e-49f9-4bee-908e-d914824af9f4","Type":"ContainerDied","Data":"25eddddae36d3a1f4ff6677f8a0785acc7f5fe642b37bd9e24cdc0b152c2f9fb"} Oct 30 20:17:12.256758 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.256737 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" event={"ID":"9f02605e-49f9-4bee-908e-d914824af9f4","Type":"ContainerDied","Data":"7ef3e8eb63145c5a45cced21577eb15ad3c7629354ec6cf9792716bf65319ebf"} Oct 30 20:17:12.256758 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.256759 4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ef3e8eb63145c5a45cced21577eb15ad3c7629354ec6cf9792716bf65319ebf" Oct 30 20:17:12.258406 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.258338 4433 generic.go:356] "Generic (PLEG): container finished" podID="6ec34b91-6b91-4031-ac85-eeb311bf4b0f" containerID="da3cc52f4a6ca4a3b65716969a978c8a07b510864e83d0e6eb636f65142be896" exitCode=0 Oct 30 20:17:12.258513 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.258410 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" event={"ID":"6ec34b91-6b91-4031-ac85-eeb311bf4b0f","Type":"ContainerDied","Data":"da3cc52f4a6ca4a3b65716969a978c8a07b510864e83d0e6eb636f65142be896"} Oct 30 20:17:12.267016 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.266947 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" Oct 30 20:17:12.312514 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.312478 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f02605e-49f9-4bee-908e-d914824af9f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f02605e-49f9-4bee-908e-d914824af9f4-plugins-config-volume\") pod \"9f02605e-49f9-4bee-908e-d914824af9f4\" (UID: \"9f02605e-49f9-4bee-908e-d914824af9f4\") " Oct 30 20:17:12.312716 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.312687 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f02605e-49f9-4bee-908e-d914824af9f4-kube-api-access-5t476\" (UniqueName: \"kubernetes.io/projected/9f02605e-49f9-4bee-908e-d914824af9f4-kube-api-access-5t476\") pod \"9f02605e-49f9-4bee-908e-d914824af9f4\" (UID: \"9f02605e-49f9-4bee-908e-d914824af9f4\") " Oct 30 20:17:12.312902 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.312880 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f02605e-49f9-4bee-908e-d914824af9f4-plugins-config-volume" pod "9f02605e-49f9-4bee-908e-d914824af9f4" (UID: "9f02605e-49f9-4bee-908e-d914824af9f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:12.313106 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.313091 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f02605e-49f9-4bee-908e-d914824af9f4-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:12.315494 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.315467 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f02605e-49f9-4bee-908e-d914824af9f4-kube-api-access-5t476" pod "9f02605e-49f9-4bee-908e-d914824af9f4" (UID: "9f02605e-49f9-4bee-908e-d914824af9f4"). InnerVolumeSpecName "kube-api-access-5t476". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:12.358471 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.358442 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" Oct 30 20:17:12.413653 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.413618 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec34b91-6b91-4031-ac85-eeb311bf4b0f-kube-api-access-q848d\" (UniqueName: \"kubernetes.io/projected/6ec34b91-6b91-4031-ac85-eeb311bf4b0f-kube-api-access-q848d\") pod \"6ec34b91-6b91-4031-ac85-eeb311bf4b0f\" (UID: \"6ec34b91-6b91-4031-ac85-eeb311bf4b0f\") " Oct 30 20:17:12.413837 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.413733 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec34b91-6b91-4031-ac85-eeb311bf4b0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec34b91-6b91-4031-ac85-eeb311bf4b0f-plugins-config-volume\") pod \"6ec34b91-6b91-4031-ac85-eeb311bf4b0f\" (UID: \"6ec34b91-6b91-4031-ac85-eeb311bf4b0f\") " Oct 30 20:17:12.414101 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.414080 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t476\" (UniqueName: \"kubernetes.io/projected/9f02605e-49f9-4bee-908e-d914824af9f4-kube-api-access-5t476\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:12.414238 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.414134 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec34b91-6b91-4031-ac85-eeb311bf4b0f-plugins-config-volume" pod "6ec34b91-6b91-4031-ac85-eeb311bf4b0f" (UID: "6ec34b91-6b91-4031-ac85-eeb311bf4b0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:12.415896 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.415866 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec34b91-6b91-4031-ac85-eeb311bf4b0f-kube-api-access-q848d" pod "6ec34b91-6b91-4031-ac85-eeb311bf4b0f" (UID: "6ec34b91-6b91-4031-ac85-eeb311bf4b0f"). InnerVolumeSpecName "kube-api-access-q848d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:12.514700 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.514668 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q848d\" (UniqueName: \"kubernetes.io/projected/6ec34b91-6b91-4031-ac85-eeb311bf4b0f-kube-api-access-q848d\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:12.514700 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:12.514694 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec34b91-6b91-4031-ac85-eeb311bf4b0f-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:17:13.265115 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:13.265085 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" Oct 30 20:17:13.265115 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:13.265100 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd" event={"ID":"6ec34b91-6b91-4031-ac85-eeb311bf4b0f","Type":"ContainerDied","Data":"02500d67e759fb556ed9874968849667d5a4f3f577d0674aa2bd5c00033a5f44"} Oct 30 20:17:13.265605 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:13.265161 4433 scope.go:122] "RemoveContainer" containerID="da3cc52f4a6ca4a3b65716969a978c8a07b510864e83d0e6eb636f65142be896" Oct 30 20:17:13.265605 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:13.265085 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258" Oct 30 20:17:13.492316 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:13.492279 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd"] Oct 30 20:17:13.530225 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:13.530144 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f1aa42-7c4579qhfd"] Oct 30 20:17:13.638189 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:13.638152 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258"] Oct 30 20:17:13.642427 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:13.642391 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f1aa42-67746f4dbhd258"] Oct 30 20:17:15.212126 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:15.212094 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec34b91-6b91-4031-ac85-eeb311bf4b0f" path="/var/lib/kubelet/pods/6ec34b91-6b91-4031-ac85-eeb311bf4b0f/volumes" Oct 30 20:17:15.212545 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:15.212474 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f02605e-49f9-4bee-908e-d914824af9f4" path="/var/lib/kubelet/pods/9f02605e-49f9-4bee-908e-d914824af9f4/volumes" Oct 30 20:17:16.184677 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:16.184642 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:16.406660 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:16.406622 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:17.570123 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:17.570085 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:23.213273 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:23.213235 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:23.366528 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:17:23.366491 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:17:44.731487 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:44.731447 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:53.366852 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:17:53.366815 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:17:56.091618 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:17:56.091579 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:23.368900 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:18:23.368804 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:18:26.521611 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:18:26.521571 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:43.018617 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:18:43.018578 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:45.077669 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:18:45.077632 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:45.307299 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:18:45.307265 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:53.366795 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:18:53.366751 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:19:23.366465 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:19:23.366425 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:19:26.862318 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:19:26.862282 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:53.367266 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:19:53.367188 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:20:01.432668 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:01.432623 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:01.974824 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:01.974785 4433 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal" podUID="d17eaea73d7bf32a8a9bb9e63451f939" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 20:20:09.869087 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:09.869053 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:10.492835 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:10.492800 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:10.731738 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:10.731704 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:14.196312 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:14.196266 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:14.441107 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:14.441055 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:17.540656 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.540608 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:17.766241 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.766202 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:17.864123 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.864096 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7"] Oct 30 20:20:17.864685 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.864660 4433 cpu_manager.go:414] "containerMap: removing container" podUID="f8038f47-7938-41c4-9fa9-9ecf844ed599" containerName="main" Oct 30 20:20:17.864685 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.864685 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8038f47-7938-41c4-9fa9-9ecf844ed599" containerName="main" Oct 30 20:20:17.864872 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.864717 4433 cpu_manager.go:414] "containerMap: removing container" podUID="6ec34b91-6b91-4031-ac85-eeb311bf4b0f" containerName="payload-pre-processing" Oct 30 20:20:17.864872 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.864725 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec34b91-6b91-4031-ac85-eeb311bf4b0f" containerName="payload-pre-processing" Oct 30 20:20:17.864872 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.864748 4433 cpu_manager.go:414] "containerMap: removing container" podUID="9f02605e-49f9-4bee-908e-d914824af9f4" containerName="payload-processing" Oct 30 20:20:17.864872 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.864756 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f02605e-49f9-4bee-908e-d914824af9f4" containerName="payload-processing" Oct 30 20:20:17.864872 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.864862 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec34b91-6b91-4031-ac85-eeb311bf4b0f" containerName="payload-pre-processing" Oct 30 20:20:17.864872 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.864873 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f02605e-49f9-4bee-908e-d914824af9f4" containerName="payload-processing" Oct 30 20:20:17.865113 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.864885 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8038f47-7938-41c4-9fa9-9ecf844ed599" containerName="main" Oct 30 20:20:17.868011 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.867974 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" Oct 30 20:20:17.971607 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.971579 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-14ab4d\"" Oct 30 20:20:17.971870 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.971852 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-3b793dfb-dockercfg-tbtxx\"" Oct 30 20:20:17.975801 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.975770 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7"] Oct 30 20:20:17.983209 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.983179 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5e7e96-11ac-487f-a8a1-0cd3a229402c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7\" (UID: \"ce5e7e96-11ac-487f-a8a1-0cd3a229402c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" Oct 30 20:20:17.983502 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:17.983480 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68jk7\" (UniqueName: \"kubernetes.io/projected/ce5e7e96-11ac-487f-a8a1-0cd3a229402c-kube-api-access-68jk7\") pod \"payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7\" (UID: \"ce5e7e96-11ac-487f-a8a1-0cd3a229402c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" Oct 30 20:20:18.023101 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.023064 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf"] Oct 30 20:20:18.026554 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.026526 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" Oct 30 20:20:18.082964 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.082933 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf"] Oct 30 20:20:18.084953 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.084929 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68jk7\" (UniqueName: \"kubernetes.io/projected/ce5e7e96-11ac-487f-a8a1-0cd3a229402c-kube-api-access-68jk7\") pod \"payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7\" (UID: \"ce5e7e96-11ac-487f-a8a1-0cd3a229402c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" Oct 30 20:20:18.085105 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.084962 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g49x\" (UniqueName: \"kubernetes.io/projected/76661b5e-a37b-431b-ac5b-c0e09b5d2778-kube-api-access-7g49x\") pod \"payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf\" (UID: \"76661b5e-a37b-431b-ac5b-c0e09b5d2778\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" Oct 30 20:20:18.085105 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.085050 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76661b5e-a37b-431b-ac5b-c0e09b5d2778-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf\" (UID: \"76661b5e-a37b-431b-ac5b-c0e09b5d2778\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" Oct 30 20:20:18.085228 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.085196 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5e7e96-11ac-487f-a8a1-0cd3a229402c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7\" (UID: \"ce5e7e96-11ac-487f-a8a1-0cd3a229402c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" Oct 30 20:20:18.085902 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.085883 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5e7e96-11ac-487f-a8a1-0cd3a229402c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7\" (UID: \"ce5e7e96-11ac-487f-a8a1-0cd3a229402c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" Oct 30 20:20:18.143456 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.143386 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68jk7\" (UniqueName: \"kubernetes.io/projected/ce5e7e96-11ac-487f-a8a1-0cd3a229402c-kube-api-access-68jk7\") pod \"payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7\" (UID: \"ce5e7e96-11ac-487f-a8a1-0cd3a229402c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" Oct 30 20:20:18.181462 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.181436 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" Oct 30 20:20:18.186716 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.186694 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g49x\" (UniqueName: \"kubernetes.io/projected/76661b5e-a37b-431b-ac5b-c0e09b5d2778-kube-api-access-7g49x\") pod \"payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf\" (UID: \"76661b5e-a37b-431b-ac5b-c0e09b5d2778\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" Oct 30 20:20:18.186816 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.186780 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76661b5e-a37b-431b-ac5b-c0e09b5d2778-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf\" (UID: \"76661b5e-a37b-431b-ac5b-c0e09b5d2778\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" Oct 30 20:20:18.187423 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.187403 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76661b5e-a37b-431b-ac5b-c0e09b5d2778-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf\" (UID: \"76661b5e-a37b-431b-ac5b-c0e09b5d2778\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" Oct 30 20:20:18.246710 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.243325 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g49x\" (UniqueName: \"kubernetes.io/projected/76661b5e-a37b-431b-ac5b-c0e09b5d2778-kube-api-access-7g49x\") pod \"payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf\" (UID: \"76661b5e-a37b-431b-ac5b-c0e09b5d2778\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" Oct 30 20:20:18.339688 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.339654 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" Oct 30 20:20:18.555528 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.555465 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7"] Oct 30 20:20:18.558260 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:20:18.558232 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5e7e96_11ac_487f_a8a1_0cd3a229402c.slice/crio-fdfe13237d10210758f447e183c58db4e69de91cce22d7ccbda858f6a5c55ef5 WatchSource:0}: Error finding container fdfe13237d10210758f447e183c58db4e69de91cce22d7ccbda858f6a5c55ef5: Status 404 returned error can't find the container with id fdfe13237d10210758f447e183c58db4e69de91cce22d7ccbda858f6a5c55ef5 Oct 30 20:20:18.713854 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:18.713755 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf"] Oct 30 20:20:18.716096 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:20:18.716072 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76661b5e_a37b_431b_ac5b_c0e09b5d2778.slice/crio-37e34fe37cfe271fcd519cdb8686bbb283cd7b95e4ad2d6b49e4baa83bf05691 WatchSource:0}: Error finding container 37e34fe37cfe271fcd519cdb8686bbb283cd7b95e4ad2d6b49e4baa83bf05691: Status 404 returned error can't find the container with id 37e34fe37cfe271fcd519cdb8686bbb283cd7b95e4ad2d6b49e4baa83bf05691 Oct 30 20:20:19.223132 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:19.223092 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" event={"ID":"76661b5e-a37b-431b-ac5b-c0e09b5d2778","Type":"ContainerStarted","Data":"18bd5fcb27b97b10e675df33dd9703e93b57aa2acea88cbdb5574d3324afbe61"} Oct 30 20:20:19.223326 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:19.223148 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" event={"ID":"76661b5e-a37b-431b-ac5b-c0e09b5d2778","Type":"ContainerStarted","Data":"37e34fe37cfe271fcd519cdb8686bbb283cd7b95e4ad2d6b49e4baa83bf05691"} Oct 30 20:20:19.223326 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:19.223192 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" Oct 30 20:20:19.224685 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:19.224657 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" event={"ID":"ce5e7e96-11ac-487f-a8a1-0cd3a229402c","Type":"ContainerStarted","Data":"8bec0d04f13a4ad78fd8a9a136fa659f27eb12b64c0ec8d704ac3aa4cf55edfa"} Oct 30 20:20:19.224861 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:19.224700 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" event={"ID":"ce5e7e96-11ac-487f-a8a1-0cd3a229402c","Type":"ContainerStarted","Data":"fdfe13237d10210758f447e183c58db4e69de91cce22d7ccbda858f6a5c55ef5"} Oct 30 20:20:19.224861 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:19.224721 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" Oct 30 20:20:19.337127 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:19.337073 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" podStartSLOduration=2.337060399 podStartE2EDuration="2.337060399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:19.33436144 +0000 UTC m=+3866.269148947" watchObservedRunningTime="2026-10-30 20:20:19.337060399 +0000 UTC m=+3866.271847923" Oct 30 20:20:19.447555 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:19.447492 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" podStartSLOduration=2.447469468 podStartE2EDuration="2.447469468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:19.446071113 +0000 UTC m=+3866.380858639" watchObservedRunningTime="2026-10-30 20:20:19.447469468 +0000 UTC m=+3866.382256995" Oct 30 20:20:20.109812 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:20.109771 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:20.327107 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:20.327056 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:20.546628 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:20.546528 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:23.366490 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:20:23.366452 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:20:25.054088 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:25.054054 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:27.193081 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:27.193020 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:29.787340 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:29.787296 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:30.236902 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:30.236875 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" Oct 30 20:20:30.237113 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:30.236933 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" Oct 30 20:20:34.405431 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:34.405377 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:39.168091 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:39.168040 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:39.407337 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:39.407302 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:39.771869 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:39.771831 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:40.006519 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:40.006480 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:42.801170 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:42.801133 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2"] Oct 30 20:20:42.805063 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:42.805042 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:42.906452 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:42.906424 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-14ab4d\"/\"openshift-service-ca.crt\"" Oct 30 20:20:42.906672 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:42.906466 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-14ab4d\"/\"kube-root-ca.crt\"" Oct 30 20:20:42.906672 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:42.906474 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-14ab4d\"/\"default-dockercfg-gs647\"" Oct 30 20:20:42.906672 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:42.906490 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-14ab4d\"/\"simulator-single-main-14ab4d-kserve-self-signed-certs\"" Oct 30 20:20:42.906882 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:42.906729 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-home\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:42.906882 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:42.906849 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5836196d-ae71-4574-8331-39895d8d7508-tls-certs\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:42.906991 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:42.906894 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmtbk\" (UniqueName: \"kubernetes.io/projected/5836196d-ae71-4574-8331-39895d8d7508-kube-api-access-cmtbk\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:42.907035 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:42.906997 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-model-cache\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:42.907080 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:42.907033 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-dshm\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:42.907116 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:42.907091 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-tmp-dir\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:42.908611 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:42.908588 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2"] Oct 30 20:20:42.986906 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:42.986862 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:43.008491 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:43.008459 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5836196d-ae71-4574-8331-39895d8d7508-tls-certs\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:43.008618 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:43.008509 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmtbk\" (UniqueName: \"kubernetes.io/projected/5836196d-ae71-4574-8331-39895d8d7508-kube-api-access-cmtbk\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:43.008618 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:43.008556 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-model-cache\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:43.008618 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:43.008572 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-dshm\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:43.008618 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:43.008600 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-tmp-dir\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:43.008858 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:43.008763 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-home\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:43.009133 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:43.009112 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-tmp-dir\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:43.009229 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:43.009130 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-model-cache\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:43.009229 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:43.009197 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-home\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:43.010750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:43.010726 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-dshm\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:43.011138 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:43.011122 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5836196d-ae71-4574-8331-39895d8d7508-tls-certs\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:43.064541 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:43.064517 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmtbk\" (UniqueName: \"kubernetes.io/projected/5836196d-ae71-4574-8331-39895d8d7508-kube-api-access-cmtbk\") pod \"simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:43.117632 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:43.117607 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:43.490376 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:43.490349 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2"] Oct 30 20:20:43.493067 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:20:43.493037 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5836196d_ae71_4574_8331_39895d8d7508.slice/crio-c844d83d06084c46832a9dc7e6b9e9bfb20737f357dc1845fa3e32beb32ac76e WatchSource:0}: Error finding container c844d83d06084c46832a9dc7e6b9e9bfb20737f357dc1845fa3e32beb32ac76e: Status 404 returned error can't find the container with id c844d83d06084c46832a9dc7e6b9e9bfb20737f357dc1845fa3e32beb32ac76e Oct 30 20:20:44.029886 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:44.029849 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:44.344890 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:44.344857 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" event={"ID":"5836196d-ae71-4574-8331-39895d8d7508","Type":"ContainerStarted","Data":"efbf831f8c1e65950768283b0f7318540970bafbeaa4de6730cc6b8d6bb811ac"} Oct 30 20:20:44.345078 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:44.344897 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" event={"ID":"5836196d-ae71-4574-8331-39895d8d7508","Type":"ContainerStarted","Data":"c844d83d06084c46832a9dc7e6b9e9bfb20737f357dc1845fa3e32beb32ac76e"} Oct 30 20:20:44.458160 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:44.458093 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" podStartSLOduration=2.458075821 podStartE2EDuration="2.458075821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:44.45707659 +0000 UTC m=+3891.391864114" watchObservedRunningTime="2026-10-30 20:20:44.458075821 +0000 UTC m=+3891.392863347" Oct 30 20:20:45.117699 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:45.117675 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:48.691801 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:48.691759 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:50.438348 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:50.438315 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:53.118487 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:53.118450 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:53.131089 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:53.131066 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:53.292564 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:53.292529 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:53.365825 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:20:53.365789 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:20:53.408847 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:53.408772 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:20:54.889376 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:54.889343 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:57.904509 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:20:57.904476 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:01.771852 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:01.771815 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:02.475509 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:02.475477 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:07.144932 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:07.144844 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:11.706489 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:11.706449 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:16.322410 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:16.322363 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:21.057115 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:21.057076 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:23.365962 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:21:23.365926 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:21:25.704731 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:25.704695 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:30.399166 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:30.399133 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:34.139886 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:34.139850 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:35.049335 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:35.049297 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:39.008105 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:39.008065 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2"] Oct 30 20:21:39.163670 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:39.163638 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:40.141048 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:40.141010 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:44.777189 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:44.777153 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:46.006255 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:46.006219 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf"] Oct 30 20:21:46.112140 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:46.112114 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7"] Oct 30 20:21:46.446753 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:46.446716 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:46.662972 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:46.662941 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:49.372689 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:49.372651 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:50.771392 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:50.771353 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:51.006521 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:51.006485 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" podUID="76661b5e-a37b-431b-ac5b-c0e09b5d2778" containerName="payload-processing" containerID="cri-o://18bd5fcb27b97b10e675df33dd9703e93b57aa2acea88cbdb5574d3324afbe61" gracePeriod=25 Oct 30 20:21:51.112644 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:51.112608 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" podUID="ce5e7e96-11ac-487f-a8a1-0cd3a229402c" containerName="payload-pre-processing" containerID="cri-o://8bec0d04f13a4ad78fd8a9a136fa659f27eb12b64c0ec8d704ac3aa4cf55edfa" gracePeriod=25 Oct 30 20:21:53.366282 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:21:53.366249 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:21:53.473473 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:53.473435 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:54.028352 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.028293 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" podUID="5836196d-ae71-4574-8331-39895d8d7508" containerName="main" containerID="cri-o://efbf831f8c1e65950768283b0f7318540970bafbeaa4de6730cc6b8d6bb811ac" gracePeriod=45 Oct 30 20:21:54.059271 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:21:54.059246 4433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5836196d_ae71_4574_8331_39895d8d7508.slice/crio-conmon-efbf831f8c1e65950768283b0f7318540970bafbeaa4de6730cc6b8d6bb811ac.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:21:54.163738 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.163713 4433 scope.go:122] "RemoveContainer" containerID="25eddddae36d3a1f4ff6677f8a0785acc7f5fe642b37bd9e24cdc0b152c2f9fb" Oct 30 20:21:54.282609 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.282556 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:21:54.363931 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.363905 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-model-cache\") pod \"5836196d-ae71-4574-8331-39895d8d7508\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " Oct 30 20:21:54.364053 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.363942 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5836196d-ae71-4574-8331-39895d8d7508-tls-certs\" (UniqueName: \"kubernetes.io/secret/5836196d-ae71-4574-8331-39895d8d7508-tls-certs\") pod \"5836196d-ae71-4574-8331-39895d8d7508\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " Oct 30 20:21:54.364053 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.364005 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5836196d-ae71-4574-8331-39895d8d7508-kube-api-access-cmtbk\" (UniqueName: \"kubernetes.io/projected/5836196d-ae71-4574-8331-39895d8d7508-kube-api-access-cmtbk\") pod \"5836196d-ae71-4574-8331-39895d8d7508\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " Oct 30 20:21:54.364173 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.364091 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-home\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-home\") pod \"5836196d-ae71-4574-8331-39895d8d7508\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " Oct 30 20:21:54.364173 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.364130 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-tmp-dir\") pod \"5836196d-ae71-4574-8331-39895d8d7508\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " Oct 30 20:21:54.364173 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.364157 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-dshm\") pod \"5836196d-ae71-4574-8331-39895d8d7508\" (UID: \"5836196d-ae71-4574-8331-39895d8d7508\") " Oct 30 20:21:54.364335 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.364174 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-model-cache" pod "5836196d-ae71-4574-8331-39895d8d7508" (UID: "5836196d-ae71-4574-8331-39895d8d7508"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:54.364424 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.364401 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-tmp-dir" pod "5836196d-ae71-4574-8331-39895d8d7508" (UID: "5836196d-ae71-4574-8331-39895d8d7508"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:54.364479 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.364404 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-home" pod "5836196d-ae71-4574-8331-39895d8d7508" (UID: "5836196d-ae71-4574-8331-39895d8d7508"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:54.364605 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.364585 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-home\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:54.364729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.364633 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-tmp-dir\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:54.364729 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.364649 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-model-cache\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:54.366244 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.366215 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-dshm" pod "5836196d-ae71-4574-8331-39895d8d7508" (UID: "5836196d-ae71-4574-8331-39895d8d7508"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:54.366244 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.366220 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5836196d-ae71-4574-8331-39895d8d7508-tls-certs" pod "5836196d-ae71-4574-8331-39895d8d7508" (UID: "5836196d-ae71-4574-8331-39895d8d7508"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:21:54.366564 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.366274 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5836196d-ae71-4574-8331-39895d8d7508-kube-api-access-cmtbk" pod "5836196d-ae71-4574-8331-39895d8d7508" (UID: "5836196d-ae71-4574-8331-39895d8d7508"). InnerVolumeSpecName "kube-api-access-cmtbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:54.465596 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.465565 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5836196d-ae71-4574-8331-39895d8d7508-dshm\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:54.465596 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.465593 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5836196d-ae71-4574-8331-39895d8d7508-tls-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:54.465777 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.465610 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmtbk\" (UniqueName: \"kubernetes.io/projected/5836196d-ae71-4574-8331-39895d8d7508-kube-api-access-cmtbk\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:21:54.707071 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.707041 4433 generic.go:356] "Generic (PLEG): container finished" podID="5836196d-ae71-4574-8331-39895d8d7508" containerID="efbf831f8c1e65950768283b0f7318540970bafbeaa4de6730cc6b8d6bb811ac" exitCode=0 Oct 30 20:21:54.707252 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.707085 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" event={"ID":"5836196d-ae71-4574-8331-39895d8d7508","Type":"ContainerDied","Data":"efbf831f8c1e65950768283b0f7318540970bafbeaa4de6730cc6b8d6bb811ac"} Oct 30 20:21:54.707252 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.707090 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" Oct 30 20:21:54.707252 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.707110 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2" event={"ID":"5836196d-ae71-4574-8331-39895d8d7508","Type":"ContainerDied","Data":"c844d83d06084c46832a9dc7e6b9e9bfb20737f357dc1845fa3e32beb32ac76e"} Oct 30 20:21:54.707252 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.707125 4433 scope.go:122] "RemoveContainer" containerID="efbf831f8c1e65950768283b0f7318540970bafbeaa4de6730cc6b8d6bb811ac" Oct 30 20:21:54.718712 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.718694 4433 scope.go:122] "RemoveContainer" containerID="efbf831f8c1e65950768283b0f7318540970bafbeaa4de6730cc6b8d6bb811ac" Oct 30 20:21:54.719012 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:21:54.718973 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efbf831f8c1e65950768283b0f7318540970bafbeaa4de6730cc6b8d6bb811ac\": container with ID starting with efbf831f8c1e65950768283b0f7318540970bafbeaa4de6730cc6b8d6bb811ac not found: ID does not exist" containerID="efbf831f8c1e65950768283b0f7318540970bafbeaa4de6730cc6b8d6bb811ac" Oct 30 20:21:54.719082 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.719025 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbf831f8c1e65950768283b0f7318540970bafbeaa4de6730cc6b8d6bb811ac"} err="failed to get container status \"efbf831f8c1e65950768283b0f7318540970bafbeaa4de6730cc6b8d6bb811ac\": rpc error: code = NotFound desc = could not find container \"efbf831f8c1e65950768283b0f7318540970bafbeaa4de6730cc6b8d6bb811ac\": container with ID starting with efbf831f8c1e65950768283b0f7318540970bafbeaa4de6730cc6b8d6bb811ac not found: ID does not exist" Oct 30 20:21:54.868732 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.868704 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2"] Oct 30 20:21:54.896269 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.896241 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:54.918046 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:54.918021 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-14ab4d/simulator-single-main-14ab4d-kserve-7b9687cb8d-w5sf2"] Oct 30 20:21:55.168016 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:55.167967 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:55.213420 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:55.213392 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5836196d-ae71-4574-8331-39895d8d7508" path="/var/lib/kubelet/pods/5836196d-ae71-4574-8331-39895d8d7508/volumes" Oct 30 20:21:55.931887 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:21:55.931852 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:00.234941 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:00.234898 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" podUID="76661b5e-a37b-431b-ac5b-c0e09b5d2778" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:22:00.235458 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:00.235333 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" podUID="ce5e7e96-11ac-487f-a8a1-0cd3a229402c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:22:02.098955 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:02.098918 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:02.731081 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:02.731049 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:03.003878 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:03.003792 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:03.258722 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:03.258639 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:10.235401 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:10.235352 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" podUID="ce5e7e96-11ac-487f-a8a1-0cd3a229402c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:22:10.235962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:10.235437 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" podUID="76661b5e-a37b-431b-ac5b-c0e09b5d2778" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:22:11.264821 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.264769 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" Oct 30 20:22:11.357216 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.357190 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" Oct 30 20:22:11.417495 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.417472 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76661b5e-a37b-431b-ac5b-c0e09b5d2778-kube-api-access-7g49x\" (UniqueName: \"kubernetes.io/projected/76661b5e-a37b-431b-ac5b-c0e09b5d2778-kube-api-access-7g49x\") pod \"76661b5e-a37b-431b-ac5b-c0e09b5d2778\" (UID: \"76661b5e-a37b-431b-ac5b-c0e09b5d2778\") " Oct 30 20:22:11.417637 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.417541 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76661b5e-a37b-431b-ac5b-c0e09b5d2778-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76661b5e-a37b-431b-ac5b-c0e09b5d2778-plugins-config-volume\") pod \"76661b5e-a37b-431b-ac5b-c0e09b5d2778\" (UID: \"76661b5e-a37b-431b-ac5b-c0e09b5d2778\") " Oct 30 20:22:11.417901 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.417876 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76661b5e-a37b-431b-ac5b-c0e09b5d2778-plugins-config-volume" pod "76661b5e-a37b-431b-ac5b-c0e09b5d2778" (UID: "76661b5e-a37b-431b-ac5b-c0e09b5d2778"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:11.419563 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.419543 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76661b5e-a37b-431b-ac5b-c0e09b5d2778-kube-api-access-7g49x" pod "76661b5e-a37b-431b-ac5b-c0e09b5d2778" (UID: "76661b5e-a37b-431b-ac5b-c0e09b5d2778"). InnerVolumeSpecName "kube-api-access-7g49x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:11.518618 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.518551 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce5e7e96-11ac-487f-a8a1-0cd3a229402c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5e7e96-11ac-487f-a8a1-0cd3a229402c-plugins-config-volume\") pod \"ce5e7e96-11ac-487f-a8a1-0cd3a229402c\" (UID: \"ce5e7e96-11ac-487f-a8a1-0cd3a229402c\") " Oct 30 20:22:11.518618 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.518585 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce5e7e96-11ac-487f-a8a1-0cd3a229402c-kube-api-access-68jk7\" (UniqueName: \"kubernetes.io/projected/ce5e7e96-11ac-487f-a8a1-0cd3a229402c-kube-api-access-68jk7\") pod \"ce5e7e96-11ac-487f-a8a1-0cd3a229402c\" (UID: \"ce5e7e96-11ac-487f-a8a1-0cd3a229402c\") " Oct 30 20:22:11.518940 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.518922 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g49x\" (UniqueName: \"kubernetes.io/projected/76661b5e-a37b-431b-ac5b-c0e09b5d2778-kube-api-access-7g49x\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:11.519020 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.518946 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76661b5e-a37b-431b-ac5b-c0e09b5d2778-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:11.519020 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.518973 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce5e7e96-11ac-487f-a8a1-0cd3a229402c-plugins-config-volume" pod "ce5e7e96-11ac-487f-a8a1-0cd3a229402c" (UID: "ce5e7e96-11ac-487f-a8a1-0cd3a229402c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:11.520604 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.520582 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5e7e96-11ac-487f-a8a1-0cd3a229402c-kube-api-access-68jk7" pod "ce5e7e96-11ac-487f-a8a1-0cd3a229402c" (UID: "ce5e7e96-11ac-487f-a8a1-0cd3a229402c"). InnerVolumeSpecName "kube-api-access-68jk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:11.620368 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.620332 4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5e7e96-11ac-487f-a8a1-0cd3a229402c-plugins-config-volume\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:11.620368 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.620361 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68jk7\" (UniqueName: \"kubernetes.io/projected/ce5e7e96-11ac-487f-a8a1-0cd3a229402c-kube-api-access-68jk7\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:22:11.798003 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.797908 4433 generic.go:356] "Generic (PLEG): container finished" podID="76661b5e-a37b-431b-ac5b-c0e09b5d2778" containerID="18bd5fcb27b97b10e675df33dd9703e93b57aa2acea88cbdb5574d3324afbe61" exitCode=0 Oct 30 20:22:11.798003 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.797954 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" Oct 30 20:22:11.798260 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.797997 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" event={"ID":"76661b5e-a37b-431b-ac5b-c0e09b5d2778","Type":"ContainerDied","Data":"18bd5fcb27b97b10e675df33dd9703e93b57aa2acea88cbdb5574d3324afbe61"} Oct 30 20:22:11.798260 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.798045 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf" event={"ID":"76661b5e-a37b-431b-ac5b-c0e09b5d2778","Type":"ContainerDied","Data":"37e34fe37cfe271fcd519cdb8686bbb283cd7b95e4ad2d6b49e4baa83bf05691"} Oct 30 20:22:11.798260 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.798072 4433 scope.go:122] "RemoveContainer" containerID="18bd5fcb27b97b10e675df33dd9703e93b57aa2acea88cbdb5574d3324afbe61" Oct 30 20:22:11.799391 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.799370 4433 generic.go:356] "Generic (PLEG): container finished" podID="ce5e7e96-11ac-487f-a8a1-0cd3a229402c" containerID="8bec0d04f13a4ad78fd8a9a136fa659f27eb12b64c0ec8d704ac3aa4cf55edfa" exitCode=0 Oct 30 20:22:11.799492 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.799404 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" Oct 30 20:22:11.799492 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.799447 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" event={"ID":"ce5e7e96-11ac-487f-a8a1-0cd3a229402c","Type":"ContainerDied","Data":"8bec0d04f13a4ad78fd8a9a136fa659f27eb12b64c0ec8d704ac3aa4cf55edfa"} Oct 30 20:22:11.799492 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.799489 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7" event={"ID":"ce5e7e96-11ac-487f-a8a1-0cd3a229402c","Type":"ContainerDied","Data":"fdfe13237d10210758f447e183c58db4e69de91cce22d7ccbda858f6a5c55ef5"} Oct 30 20:22:11.814483 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.814462 4433 scope.go:122] "RemoveContainer" containerID="18bd5fcb27b97b10e675df33dd9703e93b57aa2acea88cbdb5574d3324afbe61" Oct 30 20:22:11.814743 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:22:11.814721 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18bd5fcb27b97b10e675df33dd9703e93b57aa2acea88cbdb5574d3324afbe61\": container with ID starting with 18bd5fcb27b97b10e675df33dd9703e93b57aa2acea88cbdb5574d3324afbe61 not found: ID does not exist" containerID="18bd5fcb27b97b10e675df33dd9703e93b57aa2acea88cbdb5574d3324afbe61" Oct 30 20:22:11.814801 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.814750 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18bd5fcb27b97b10e675df33dd9703e93b57aa2acea88cbdb5574d3324afbe61"} err="failed to get container status \"18bd5fcb27b97b10e675df33dd9703e93b57aa2acea88cbdb5574d3324afbe61\": rpc error: code = NotFound desc = could not find container \"18bd5fcb27b97b10e675df33dd9703e93b57aa2acea88cbdb5574d3324afbe61\": container with ID starting with 18bd5fcb27b97b10e675df33dd9703e93b57aa2acea88cbdb5574d3324afbe61 not found: ID does not exist" Oct 30 20:22:11.814801 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.814772 4433 scope.go:122] "RemoveContainer" containerID="8bec0d04f13a4ad78fd8a9a136fa659f27eb12b64c0ec8d704ac3aa4cf55edfa" Oct 30 20:22:11.823383 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.823361 4433 scope.go:122] "RemoveContainer" containerID="8bec0d04f13a4ad78fd8a9a136fa659f27eb12b64c0ec8d704ac3aa4cf55edfa" Oct 30 20:22:11.823653 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:22:11.823632 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bec0d04f13a4ad78fd8a9a136fa659f27eb12b64c0ec8d704ac3aa4cf55edfa\": container with ID starting with 8bec0d04f13a4ad78fd8a9a136fa659f27eb12b64c0ec8d704ac3aa4cf55edfa not found: ID does not exist" containerID="8bec0d04f13a4ad78fd8a9a136fa659f27eb12b64c0ec8d704ac3aa4cf55edfa" Oct 30 20:22:11.823708 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.823659 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bec0d04f13a4ad78fd8a9a136fa659f27eb12b64c0ec8d704ac3aa4cf55edfa"} err="failed to get container status \"8bec0d04f13a4ad78fd8a9a136fa659f27eb12b64c0ec8d704ac3aa4cf55edfa\": rpc error: code = NotFound desc = could not find container \"8bec0d04f13a4ad78fd8a9a136fa659f27eb12b64c0ec8d704ac3aa4cf55edfa\": container with ID starting with 8bec0d04f13a4ad78fd8a9a136fa659f27eb12b64c0ec8d704ac3aa4cf55edfa not found: ID does not exist" Oct 30 20:22:11.960142 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:11.960109 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7"] Oct 30 20:22:12.009533 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:12.009496 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-14ab4d-84fbcsnfj7"] Oct 30 20:22:12.072299 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:12.072268 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf"] Oct 30 20:22:12.077938 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:12.077914 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-14ab4d-8598d64bb45kvf"] Oct 30 20:22:13.212366 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:13.212331 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76661b5e-a37b-431b-ac5b-c0e09b5d2778" path="/var/lib/kubelet/pods/76661b5e-a37b-431b-ac5b-c0e09b5d2778/volumes" Oct 30 20:22:13.212856 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:13.212750 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce5e7e96-11ac-487f-a8a1-0cd3a229402c" path="/var/lib/kubelet/pods/ce5e7e96-11ac-487f-a8a1-0cd3a229402c/volumes" Oct 30 20:22:15.278355 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:15.278318 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:16.545079 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:16.545039 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:22.900633 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:22.900590 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:23.366795 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:22:23.366763 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:22:32.543382 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:32.543337 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:35.093645 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:35.093564 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:51.120974 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:51.120937 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:53.366053 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:22:53.366009 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:22:58.395596 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.395560 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg"] Oct 30 20:22:58.396035 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.395947 4433 cpu_manager.go:414] "containerMap: removing container" podUID="ce5e7e96-11ac-487f-a8a1-0cd3a229402c" containerName="payload-pre-processing" Oct 30 20:22:58.396035 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.395957 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5e7e96-11ac-487f-a8a1-0cd3a229402c" containerName="payload-pre-processing" Oct 30 20:22:58.396035 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.395973 4433 cpu_manager.go:414] "containerMap: removing container" podUID="5836196d-ae71-4574-8331-39895d8d7508" containerName="main" Oct 30 20:22:58.396035 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.395992 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5836196d-ae71-4574-8331-39895d8d7508" containerName="main" Oct 30 20:22:58.396035 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.396015 4433 cpu_manager.go:414] "containerMap: removing container" podUID="76661b5e-a37b-431b-ac5b-c0e09b5d2778" containerName="payload-processing" Oct 30 20:22:58.396035 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.396020 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76661b5e-a37b-431b-ac5b-c0e09b5d2778" containerName="payload-processing" Oct 30 20:22:58.396290 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.396083 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5e7e96-11ac-487f-a8a1-0cd3a229402c" containerName="payload-pre-processing" Oct 30 20:22:58.396290 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.396093 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5836196d-ae71-4574-8331-39895d8d7508" containerName="main" Oct 30 20:22:58.396290 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.396102 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="76661b5e-a37b-431b-ac5b-c0e09b5d2778" containerName="payload-processing" Oct 30 20:22:58.400515 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.400495 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.504135 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.504103 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-68b4cd44-kserve-self-signed-certs\"" Oct 30 20:22:58.506059 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.506034 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg"] Oct 30 20:22:58.527054 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.527023 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r96rm\" (UniqueName: \"kubernetes.io/projected/e69fadbe-3be2-4f93-a484-698f54b8842e-kube-api-access-r96rm\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.527192 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.527070 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69fadbe-3be2-4f93-a484-698f54b8842e-tls-certs\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.527237 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.527182 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-model-cache\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.527237 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.527215 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-dshm\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.527326 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.527311 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-tmp-dir\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.527376 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.527362 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-home\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.572888 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.572850 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:58.628058 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.628026 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-model-cache\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.628058 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.628063 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-dshm\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.628273 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.628105 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-tmp-dir\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.628273 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.628150 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-home\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.628273 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.628171 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96rm\" (UniqueName: \"kubernetes.io/projected/e69fadbe-3be2-4f93-a484-698f54b8842e-kube-api-access-r96rm\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.628417 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.628292 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69fadbe-3be2-4f93-a484-698f54b8842e-tls-certs\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.628504 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.628484 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-model-cache\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.628592 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.628573 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-tmp-dir\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.628650 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.628623 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-home\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.630365 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.630341 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-dshm\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.630651 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.630632 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69fadbe-3be2-4f93-a484-698f54b8842e-tls-certs\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.684298 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.684235 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r96rm\" (UniqueName: \"kubernetes.io/projected/e69fadbe-3be2-4f93-a484-698f54b8842e-kube-api-access-r96rm\") pod \"e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:58.711817 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:58.711788 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:22:59.082294 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:22:59.082274 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg"] Oct 30 20:22:59.084813 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:22:59.084787 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode69fadbe_3be2_4f93_a484_698f54b8842e.slice/crio-ad605be6f4b83076ecdcd2e324a8247cff4ea91c546a24ab0cd0a5ca269a9615 WatchSource:0}: Error finding container ad605be6f4b83076ecdcd2e324a8247cff4ea91c546a24ab0cd0a5ca269a9615: Status 404 returned error can't find the container with id ad605be6f4b83076ecdcd2e324a8247cff4ea91c546a24ab0cd0a5ca269a9615 Oct 30 20:23:00.061821 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:00.061783 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" event={"ID":"e69fadbe-3be2-4f93-a484-698f54b8842e","Type":"ContainerStarted","Data":"1f9e5e1b3eceac9fe9f42ba0c764fb094a8b3d11608457c2780bc9152c4f9a3f"} Oct 30 20:23:00.062233 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:00.061837 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" event={"ID":"e69fadbe-3be2-4f93-a484-698f54b8842e","Type":"ContainerStarted","Data":"ad605be6f4b83076ecdcd2e324a8247cff4ea91c546a24ab0cd0a5ca269a9615"} Oct 30 20:23:00.178362 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:00.178300 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" podStartSLOduration=2.178280348 podStartE2EDuration="2.178280348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:00.175027295 +0000 UTC m=+4027.109814815" watchObservedRunningTime="2026-10-30 20:23:00.178280348 +0000 UTC m=+4027.113067873" Oct 30 20:23:00.712213 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:00.712180 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:23:02.450906 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:02.450872 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:06.005165 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:06.005125 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gtk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:23:06.005818 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:06.005781 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:23:08.712261 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:08.712220 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:23:08.724489 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:08.724463 4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:23:09.130674 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:09.130644 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:23:23.365294 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:23:23.365259 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:23:32.910203 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:32.910161 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:44.119915 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:44.119876 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:51.005144 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:51.005098 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gtk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:23:51.005656 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:51.005178 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:23:51.462000 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:51.461949 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:53.364953 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:23:53.364922 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:23:59.212709 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:59.212675 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg"] Oct 30 20:23:59.322180 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:23:59.322118 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:02.787958 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:02.787921 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:06.016057 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:06.015074 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r7gtk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:24:06.016057 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:06.015208 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:24:11.501175 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:11.501116 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:14.231362 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.231311 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" podUID="e69fadbe-3be2-4f93-a484-698f54b8842e" containerName="main" containerID="cri-o://1f9e5e1b3eceac9fe9f42ba0c764fb094a8b3d11608457c2780bc9152c4f9a3f" gracePeriod=45 Oct 30 20:24:14.483827 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.483759 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:24:14.519932 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.519903 4433 generic.go:356] "Generic (PLEG): container finished" podID="e69fadbe-3be2-4f93-a484-698f54b8842e" containerID="1f9e5e1b3eceac9fe9f42ba0c764fb094a8b3d11608457c2780bc9152c4f9a3f" exitCode=0 Oct 30 20:24:14.520077 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.519937 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" Oct 30 20:24:14.520077 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.519931 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" event={"ID":"e69fadbe-3be2-4f93-a484-698f54b8842e","Type":"ContainerDied","Data":"1f9e5e1b3eceac9fe9f42ba0c764fb094a8b3d11608457c2780bc9152c4f9a3f"} Oct 30 20:24:14.520077 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.520027 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg" event={"ID":"e69fadbe-3be2-4f93-a484-698f54b8842e","Type":"ContainerDied","Data":"ad605be6f4b83076ecdcd2e324a8247cff4ea91c546a24ab0cd0a5ca269a9615"} Oct 30 20:24:14.520077 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.520044 4433 scope.go:122] "RemoveContainer" containerID="1f9e5e1b3eceac9fe9f42ba0c764fb094a8b3d11608457c2780bc9152c4f9a3f" Oct 30 20:24:14.528492 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.528475 4433 scope.go:122] "RemoveContainer" containerID="1f9e5e1b3eceac9fe9f42ba0c764fb094a8b3d11608457c2780bc9152c4f9a3f" Oct 30 20:24:14.528734 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:24:14.528709 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9e5e1b3eceac9fe9f42ba0c764fb094a8b3d11608457c2780bc9152c4f9a3f\": container with ID starting with 1f9e5e1b3eceac9fe9f42ba0c764fb094a8b3d11608457c2780bc9152c4f9a3f not found: ID does not exist" containerID="1f9e5e1b3eceac9fe9f42ba0c764fb094a8b3d11608457c2780bc9152c4f9a3f" Oct 30 20:24:14.528790 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.528744 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9e5e1b3eceac9fe9f42ba0c764fb094a8b3d11608457c2780bc9152c4f9a3f"} err="failed to get container status \"1f9e5e1b3eceac9fe9f42ba0c764fb094a8b3d11608457c2780bc9152c4f9a3f\": rpc error: code = NotFound desc = could not find container \"1f9e5e1b3eceac9fe9f42ba0c764fb094a8b3d11608457c2780bc9152c4f9a3f\": container with ID starting with 1f9e5e1b3eceac9fe9f42ba0c764fb094a8b3d11608457c2780bc9152c4f9a3f not found: ID does not exist" Oct 30 20:24:14.596383 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.596363 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69fadbe-3be2-4f93-a484-698f54b8842e-kube-api-access-r96rm\" (UniqueName: \"kubernetes.io/projected/e69fadbe-3be2-4f93-a484-698f54b8842e-kube-api-access-r96rm\") pod \"e69fadbe-3be2-4f93-a484-698f54b8842e\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " Oct 30 20:24:14.596480 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.596419 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e69fadbe-3be2-4f93-a484-698f54b8842e-tls-certs\" (UniqueName: \"kubernetes.io/secret/e69fadbe-3be2-4f93-a484-698f54b8842e-tls-certs\") pod \"e69fadbe-3be2-4f93-a484-698f54b8842e\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " Oct 30 20:24:14.596480 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.596447 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-dshm\") pod \"e69fadbe-3be2-4f93-a484-698f54b8842e\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " Oct 30 20:24:14.596573 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.596516 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-model-cache\") pod \"e69fadbe-3be2-4f93-a484-698f54b8842e\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " Oct 30 20:24:14.596630 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.596613 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-home\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-home\") pod \"e69fadbe-3be2-4f93-a484-698f54b8842e\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " Oct 30 20:24:14.596681 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.596651 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-tmp-dir\") pod \"e69fadbe-3be2-4f93-a484-698f54b8842e\" (UID: \"e69fadbe-3be2-4f93-a484-698f54b8842e\") " Oct 30 20:24:14.596899 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.596848 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-model-cache" pod "e69fadbe-3be2-4f93-a484-698f54b8842e" (UID: "e69fadbe-3be2-4f93-a484-698f54b8842e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:14.597031 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.596964 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-home" pod "e69fadbe-3be2-4f93-a484-698f54b8842e" (UID: "e69fadbe-3be2-4f93-a484-698f54b8842e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:14.597105 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.597065 4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-model-cache\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:24:14.597105 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.597075 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-tmp-dir" pod "e69fadbe-3be2-4f93-a484-698f54b8842e" (UID: "e69fadbe-3be2-4f93-a484-698f54b8842e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:14.599354 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.599320 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69fadbe-3be2-4f93-a484-698f54b8842e-tls-certs" pod "e69fadbe-3be2-4f93-a484-698f54b8842e" (UID: "e69fadbe-3be2-4f93-a484-698f54b8842e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:24:14.599449 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.599332 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-dshm" pod "e69fadbe-3be2-4f93-a484-698f54b8842e" (UID: "e69fadbe-3be2-4f93-a484-698f54b8842e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:14.599449 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.599434 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69fadbe-3be2-4f93-a484-698f54b8842e-kube-api-access-r96rm" pod "e69fadbe-3be2-4f93-a484-698f54b8842e" (UID: "e69fadbe-3be2-4f93-a484-698f54b8842e"). InnerVolumeSpecName "kube-api-access-r96rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:14.698107 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.698079 4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-home\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:24:14.698107 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.698104 4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-tmp-dir\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:24:14.698244 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.698120 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r96rm\" (UniqueName: \"kubernetes.io/projected/e69fadbe-3be2-4f93-a484-698f54b8842e-kube-api-access-r96rm\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:24:14.698244 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.698133 4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69fadbe-3be2-4f93-a484-698f54b8842e-tls-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:24:14.698244 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.698144 4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69fadbe-3be2-4f93-a484-698f54b8842e-dshm\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:24:14.937728 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.937704 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg"] Oct 30 20:24:14.943498 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:14.943477 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-68b4cd44-kserve-6bbc6c6bcc-xllbg"] Oct 30 20:24:15.212420 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:15.212360 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e69fadbe-3be2-4f93-a484-698f54b8842e" path="/var/lib/kubelet/pods/e69fadbe-3be2-4f93-a484-698f54b8842e/volumes" Oct 30 20:24:15.776995 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:15.776945 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:23.364324 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:24:23.364286 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:24:33.310319 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:33.310277 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:51.801484 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:51.801451 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:53.364878 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:24:53.364840 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:24:59.962030 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:59.961958 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-4j76p"] Oct 30 20:24:59.962459 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:59.962449 4433 cpu_manager.go:414] "containerMap: removing container" podUID="e69fadbe-3be2-4f93-a484-698f54b8842e" containerName="main" Oct 30 20:24:59.962526 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:59.962464 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e69fadbe-3be2-4f93-a484-698f54b8842e" containerName="main" Oct 30 20:24:59.962566 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:59.962550 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e69fadbe-3be2-4f93-a484-698f54b8842e" containerName="main" Oct 30 20:24:59.966912 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:24:59.966892 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" Oct 30 20:25:00.063141 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.063107 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59-openshift-service-ca\") pod \"maas-api-595cc648c-4j76p\" (UID: \"371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" Oct 30 20:25:00.063275 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.063154 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdxps\" (UniqueName: \"kubernetes.io/projected/371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59-kube-api-access-qdxps\") pod \"maas-api-595cc648c-4j76p\" (UID: \"371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" Oct 30 20:25:00.063275 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.063242 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59-maas-api-tls\") pod \"maas-api-595cc648c-4j76p\" (UID: \"371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" Oct 30 20:25:00.063354 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.063336 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59-metrics-certs\") pod \"maas-api-595cc648c-4j76p\" (UID: \"371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" Oct 30 20:25:00.069570 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.069551 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-4j76p"] Oct 30 20:25:00.164252 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.164228 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59-metrics-certs\") pod \"maas-api-595cc648c-4j76p\" (UID: \"371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" Oct 30 20:25:00.164410 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.164302 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59-openshift-service-ca\") pod \"maas-api-595cc648c-4j76p\" (UID: \"371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" Oct 30 20:25:00.164410 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.164336 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdxps\" (UniqueName: \"kubernetes.io/projected/371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59-kube-api-access-qdxps\") pod \"maas-api-595cc648c-4j76p\" (UID: \"371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" Oct 30 20:25:00.164410 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.164379 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59-maas-api-tls\") pod \"maas-api-595cc648c-4j76p\" (UID: \"371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" Oct 30 20:25:00.164964 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.164943 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59-openshift-service-ca\") pod \"maas-api-595cc648c-4j76p\" (UID: \"371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" Oct 30 20:25:00.166788 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.166762 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59-maas-api-tls\") pod \"maas-api-595cc648c-4j76p\" (UID: \"371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" Oct 30 20:25:00.166890 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.166768 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59-metrics-certs\") pod \"maas-api-595cc648c-4j76p\" (UID: \"371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" Oct 30 20:25:00.222548 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.222492 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdxps\" (UniqueName: \"kubernetes.io/projected/371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59-kube-api-access-qdxps\") pod \"maas-api-595cc648c-4j76p\" (UID: \"371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" Oct 30 20:25:00.279694 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.279671 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" Oct 30 20:25:00.654806 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.654777 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-4j76p"] Oct 30 20:25:00.657145 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:25:00.657117 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371a5ee5_0ee2_4092_8a6c_aa8dd1f43a59.slice/crio-3a8b3586eb4822ace7d01bcef982dc05f26c4b016d15a65a5e3da2ea5590dfe4 WatchSource:0}: Error finding container 3a8b3586eb4822ace7d01bcef982dc05f26c4b016d15a65a5e3da2ea5590dfe4: Status 404 returned error can't find the container with id 3a8b3586eb4822ace7d01bcef982dc05f26c4b016d15a65a5e3da2ea5590dfe4 Oct 30 20:25:00.658940 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.658919 4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:25:00.805308 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:00.805265 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" event={"ID":"371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59","Type":"ContainerStarted","Data":"3a8b3586eb4822ace7d01bcef982dc05f26c4b016d15a65a5e3da2ea5590dfe4"} Oct 30 20:25:01.814181 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:01.814141 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" event={"ID":"371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59","Type":"ContainerStarted","Data":"e42a03248aa9d14be42e869287f57ea2b29a0d8c05e4bf8bf3859b1914874390"} Oct 30 20:25:01.814683 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:01.814444 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" Oct 30 20:25:01.932015 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:01.931872 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" podStartSLOduration=1.9551986160000001 podStartE2EDuration="2.93185324s" totalImagesPullingTime="976.654624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:01.929166945 +0000 UTC m=+4148.863954473" watchObservedRunningTime="2026-10-30 20:25:01.93185324 +0000 UTC m=+4148.866640764" Oct 30 20:25:07.842177 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:07.842146 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-4j76p" Oct 30 20:25:08.006834 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:08.006802 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2"] Oct 30 20:25:08.007080 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:08.007050 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" podUID="860c3b62-aa0e-466b-9991-029a5cf5a394" containerName="maas-api" containerID="cri-o://1ac50935d0e20a623605665b66d5a2dfd64340b8d0c35507268cc794b7b5057c" gracePeriod=30 Oct 30 20:25:09.272740 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.272715 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 20:25:09.400779 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.400711 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/860c3b62-aa0e-466b-9991-029a5cf5a394-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860c3b62-aa0e-466b-9991-029a5cf5a394-openshift-service-ca\") pod \"860c3b62-aa0e-466b-9991-029a5cf5a394\" (UID: \"860c3b62-aa0e-466b-9991-029a5cf5a394\") " Oct 30 20:25:09.400895 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.400843 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860c3b62-aa0e-466b-9991-029a5cf5a394-kube-api-access-hcn99\" (UniqueName: \"kubernetes.io/projected/860c3b62-aa0e-466b-9991-029a5cf5a394-kube-api-access-hcn99\") pod \"860c3b62-aa0e-466b-9991-029a5cf5a394\" (UID: \"860c3b62-aa0e-466b-9991-029a5cf5a394\") " Oct 30 20:25:09.400895 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.400887 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/860c3b62-aa0e-466b-9991-029a5cf5a394-metrics-certs\" (UniqueName: \"kubernetes.io/secret/860c3b62-aa0e-466b-9991-029a5cf5a394-metrics-certs\") pod \"860c3b62-aa0e-466b-9991-029a5cf5a394\" (UID: \"860c3b62-aa0e-466b-9991-029a5cf5a394\") " Oct 30 20:25:09.401077 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.401051 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/860c3b62-aa0e-466b-9991-029a5cf5a394-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860c3b62-aa0e-466b-9991-029a5cf5a394-maas-api-tls\") pod \"860c3b62-aa0e-466b-9991-029a5cf5a394\" (UID: \"860c3b62-aa0e-466b-9991-029a5cf5a394\") " Oct 30 20:25:09.401143 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.401093 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860c3b62-aa0e-466b-9991-029a5cf5a394-openshift-service-ca" pod "860c3b62-aa0e-466b-9991-029a5cf5a394" (UID: "860c3b62-aa0e-466b-9991-029a5cf5a394"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:09.401441 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.401408 4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860c3b62-aa0e-466b-9991-029a5cf5a394-openshift-service-ca\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:25:09.402994 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.402953 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/860c3b62-aa0e-466b-9991-029a5cf5a394-metrics-certs" pod "860c3b62-aa0e-466b-9991-029a5cf5a394" (UID: "860c3b62-aa0e-466b-9991-029a5cf5a394"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:25:09.403399 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.403376 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/860c3b62-aa0e-466b-9991-029a5cf5a394-maas-api-tls" pod "860c3b62-aa0e-466b-9991-029a5cf5a394" (UID: "860c3b62-aa0e-466b-9991-029a5cf5a394"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:25:09.403476 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.403411 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860c3b62-aa0e-466b-9991-029a5cf5a394-kube-api-access-hcn99" pod "860c3b62-aa0e-466b-9991-029a5cf5a394" (UID: "860c3b62-aa0e-466b-9991-029a5cf5a394"). InnerVolumeSpecName "kube-api-access-hcn99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:09.502020 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.501999 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcn99\" (UniqueName: \"kubernetes.io/projected/860c3b62-aa0e-466b-9991-029a5cf5a394-kube-api-access-hcn99\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:25:09.502020 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.502021 4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/860c3b62-aa0e-466b-9991-029a5cf5a394-metrics-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:25:09.502148 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.502032 4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860c3b62-aa0e-466b-9991-029a5cf5a394-maas-api-tls\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:25:09.883152 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.883129 4433 generic.go:356] "Generic (PLEG): container finished" podID="860c3b62-aa0e-466b-9991-029a5cf5a394" containerID="1ac50935d0e20a623605665b66d5a2dfd64340b8d0c35507268cc794b7b5057c" exitCode=0 Oct 30 20:25:09.883268 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.883177 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" Oct 30 20:25:09.883268 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.883195 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" event={"ID":"860c3b62-aa0e-466b-9991-029a5cf5a394","Type":"ContainerDied","Data":"1ac50935d0e20a623605665b66d5a2dfd64340b8d0c35507268cc794b7b5057c"} Oct 30 20:25:09.883268 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.883232 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2" event={"ID":"860c3b62-aa0e-466b-9991-029a5cf5a394","Type":"ContainerDied","Data":"b7c14cac48a8db2330d6df265c3304a4893a6c2b131c401947f26e221b26fef7"} Oct 30 20:25:09.883268 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.883253 4433 scope.go:122] "RemoveContainer" containerID="1ac50935d0e20a623605665b66d5a2dfd64340b8d0c35507268cc794b7b5057c" Oct 30 20:25:09.893006 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.892974 4433 scope.go:122] "RemoveContainer" containerID="e46b255da168032a2f553a172d19f7ed857ec4f4c39351e9c85e2c01a2bb99aa" Oct 30 20:25:09.900748 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.900734 4433 scope.go:122] "RemoveContainer" containerID="1ac50935d0e20a623605665b66d5a2dfd64340b8d0c35507268cc794b7b5057c" Oct 30 20:25:09.901006 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:25:09.900973 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac50935d0e20a623605665b66d5a2dfd64340b8d0c35507268cc794b7b5057c\": container with ID starting with 1ac50935d0e20a623605665b66d5a2dfd64340b8d0c35507268cc794b7b5057c not found: ID does not exist" containerID="1ac50935d0e20a623605665b66d5a2dfd64340b8d0c35507268cc794b7b5057c" Oct 30 20:25:09.901066 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.901013 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac50935d0e20a623605665b66d5a2dfd64340b8d0c35507268cc794b7b5057c"} err="failed to get container status \"1ac50935d0e20a623605665b66d5a2dfd64340b8d0c35507268cc794b7b5057c\": rpc error: code = NotFound desc = could not find container \"1ac50935d0e20a623605665b66d5a2dfd64340b8d0c35507268cc794b7b5057c\": container with ID starting with 1ac50935d0e20a623605665b66d5a2dfd64340b8d0c35507268cc794b7b5057c not found: ID does not exist" Oct 30 20:25:09.901066 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.901029 4433 scope.go:122] "RemoveContainer" containerID="e46b255da168032a2f553a172d19f7ed857ec4f4c39351e9c85e2c01a2bb99aa" Oct 30 20:25:09.901254 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:25:09.901237 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e46b255da168032a2f553a172d19f7ed857ec4f4c39351e9c85e2c01a2bb99aa\": container with ID starting with e46b255da168032a2f553a172d19f7ed857ec4f4c39351e9c85e2c01a2bb99aa not found: ID does not exist" containerID="e46b255da168032a2f553a172d19f7ed857ec4f4c39351e9c85e2c01a2bb99aa" Oct 30 20:25:09.901293 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:09.901262 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46b255da168032a2f553a172d19f7ed857ec4f4c39351e9c85e2c01a2bb99aa"} err="failed to get container status \"e46b255da168032a2f553a172d19f7ed857ec4f4c39351e9c85e2c01a2bb99aa\": rpc error: code = NotFound desc = could not find container \"e46b255da168032a2f553a172d19f7ed857ec4f4c39351e9c85e2c01a2bb99aa\": container with ID starting with e46b255da168032a2f553a172d19f7ed857ec4f4c39351e9c85e2c01a2bb99aa not found: ID does not exist" Oct 30 20:25:10.075560 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:10.075538 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2"] Oct 30 20:25:10.124858 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:10.124836 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hz2x2"] Oct 30 20:25:11.212382 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:11.212348 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860c3b62-aa0e-466b-9991-029a5cf5a394" path="/var/lib/kubelet/pods/860c3b62-aa0e-466b-9991-029a5cf5a394/volumes" Oct 30 20:25:16.033931 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:16.033899 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:16.250210 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:16.250180 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:23.364224 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:25:23.364186 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:25:43.624008 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:43.623887 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:52.125396 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:25:52.125360 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:53.364893 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:25:53.364852 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:26:08.137819 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:26:08.137784 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:23.364733 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:26:23.364701 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:26:30.461508 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:26:30.461475 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cb7567d84-s4vn7_f9b94090-336f-434e-a889-a9cb0b8f571c/authorino/0.log" Oct 30 20:26:43.968381 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:26:43.968351 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:51.233396 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:26:51.233364 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-76877f96df-kbpcw_adeb74f6-f548-4164-865b-0b8a2d1948b9/manager/0.log" Oct 30 20:26:51.607141 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:26:51.607115 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-tzbdd_2959b203-6fb1-492c-bcbf-c1034896b4cf/manager/0.log" Oct 30 20:26:51.839776 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:26:51.839753 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-84kt5_f8970bfd-bb0e-4266-92f4-4313f641b671/manager/0.log" Oct 30 20:26:52.377598 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:26:52.377571 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:53.365144 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:26:53.365113 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:26:54.250014 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:26:54.249990 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-299ch_c28650ad-4927-4854-a511-4b8279b97e33/manager/0.log" Oct 30 20:26:54.676844 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:26:54.676814 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69bfcc8945-6f4hg_5c5a1734-680c-472c-a544-723789285116/manager/1.log" Oct 30 20:26:54.742993 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:26:54.742959 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69bfcc8945-6f4hg_5c5a1734-680c-472c-a544-723789285116/copy-manifests/0.log" Oct 30 20:26:56.262474 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:26:56.262443 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:56.305179 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:26:56.305153 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69bfcc8945-6f4hg_5c5a1734-680c-472c-a544-723789285116/manager/0.log" Oct 30 20:26:56.915517 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:26:56.915490 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-4j76p_371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59/maas-api/0.log" Oct 30 20:26:57.066465 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:26:57.066436 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:00.405941 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:00.405912 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cb7567d84-s4vn7_f9b94090-336f-434e-a889-a9cb0b8f571c/authorino/0.log" Oct 30 20:27:01.419538 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:01.419511 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5drm6_67eba015-5cff-4ff1-b851-3f53d4497847/kuadrant-console-plugin/0.log" Oct 30 20:27:02.424573 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:02.424545 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-gnbjg_766d6d65-943d-4ae0-9c0c-1601453ff404/manager/0.log" Oct 30 20:27:13.364962 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:13.364888 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5xf482_40b46873-539f-42cb-856a-f28941276775/istio-proxy/0.log" Oct 30 20:27:15.343799 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:15.343770 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-fzmkt_3485be44-cb71-42c4-ab8e-dfcd161784cd/istio-proxy/0.log" Oct 30 20:27:15.949688 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:15.949660 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-ld68z_485179f9-ab6f-4ff6-8520-437297c69df5/istio-proxy/0.log" Oct 30 20:27:16.826351 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:16.826325 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-pmt9x_f7f198b8-0ce4-4ac4-9339-944e7f829bf1/istio-proxy/0.log" Oct 30 20:27:17.106160 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:17.106099 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r7gtk_ef5e46b2-1008-4306-92fb-abfae43bcd79/istio-proxy/0.log" Oct 30 20:27:17.394924 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:17.394822 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-t2t65_f8a5fdf4-c366-4af9-b351-15a6617824d7/payload-pre-processing/0.log" Oct 30 20:27:17.644675 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:17.644652 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-69g25_da86486a-f4f9-4bc3-a5e0-be4073682c0c/payload-processing/0.log" Oct 30 20:27:23.365360 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:27:23.365325 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:27:29.071167 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:29.071142 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-f6zfs_da30980c-5b69-448d-a0c3-b2e21f677d8f/main/0.log" Oct 30 20:27:29.617726 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:29.617703 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-x652r_51dce2ca-7adf-4e00-8d76-c32a82932883/main/0.log" Oct 30 20:27:30.455730 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:30.455701 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk_ee9f5fbf-a71a-491e-abd3-8b59a15066ae/main/0.log" Oct 30 20:27:31.445976 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:31.445947 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cb7567d84-s4vn7_f9b94090-336f-434e-a889-a9cb0b8f571c/authorino/0.log" Oct 30 20:27:44.301643 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:44.301609 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:52.441148 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:52.441115 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:52.678539 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:52.678502 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:52.957520 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:27:52.957493 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:53.366018 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:27:53.365989 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:28:00.766346 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:00.766320 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-76877f96df-kbpcw_adeb74f6-f548-4164-865b-0b8a2d1948b9/manager/0.log" Oct 30 20:28:01.146083 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:01.146055 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-tzbdd_2959b203-6fb1-492c-bcbf-c1034896b4cf/manager/0.log" Oct 30 20:28:01.377543 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:01.377521 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-84kt5_f8970bfd-bb0e-4266-92f4-4313f641b671/manager/0.log" Oct 30 20:28:03.004248 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:03.004178 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-299ch_c28650ad-4927-4854-a511-4b8279b97e33/manager/0.log" Oct 30 20:28:03.447371 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:03.447349 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69bfcc8945-6f4hg_5c5a1734-680c-472c-a544-723789285116/copy-manifests/0.log" Oct 30 20:28:03.516294 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:03.516271 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69bfcc8945-6f4hg_5c5a1734-680c-472c-a544-723789285116/manager/1.log" Oct 30 20:28:13.479851 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:13.479810 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:15.240578 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:15.240549 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-4j76p_371a5ee5-0ee2-4092-8a6c-aa8dd1f43a59/maas-api/0.log" Oct 30 20:28:23.365677 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:28:23.365642 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:28:28.478801 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:28.478769 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cb7567d84-s4vn7_f9b94090-336f-434e-a889-a9cb0b8f571c/authorino/0.log" Oct 30 20:28:29.488587 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:29.488559 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5drm6_67eba015-5cff-4ff1-b851-3f53d4497847/kuadrant-console-plugin/0.log" Oct 30 20:28:30.417279 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:30.417253 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-gnbjg_766d6d65-943d-4ae0-9c0c-1601453ff404/manager/0.log" Oct 30 20:28:31.126192 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:31.126165 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5xf482_40b46873-539f-42cb-856a-f28941276775/istio-proxy/0.log" Oct 30 20:28:32.933904 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:32.933878 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-fzmkt_3485be44-cb71-42c4-ab8e-dfcd161784cd/istio-proxy/0.log" Oct 30 20:28:33.544552 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:33.544531 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-ld68z_485179f9-ab6f-4ff6-8520-437297c69df5/istio-proxy/0.log" Oct 30 20:28:34.347619 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:34.347585 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-pmt9x_f7f198b8-0ce4-4ac4-9339-944e7f829bf1/istio-proxy/0.log" Oct 30 20:28:34.623010 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:34.622942 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r7gtk_ef5e46b2-1008-4306-92fb-abfae43bcd79/istio-proxy/0.log" Oct 30 20:28:34.898512 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:34.898421 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-t2t65_f8a5fdf4-c366-4af9-b351-15a6617824d7/payload-pre-processing/0.log" Oct 30 20:28:35.121738 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:35.121719 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-69g25_da86486a-f4f9-4bc3-a5e0-be4073682c0c/payload-processing/0.log" Oct 30 20:28:36.304906 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:36.304882 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-f6zfs_da30980c-5b69-448d-a0c3-b2e21f677d8f/main/0.log" Oct 30 20:28:36.852235 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:36.852192 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-x652r_51dce2ca-7adf-4e00-8d76-c32a82932883/main/0.log" Oct 30 20:28:37.650738 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:37.650714 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-gs6mk_ee9f5fbf-a71a-491e-abd3-8b59a15066ae/main/0.log" Oct 30 20:28:44.643293 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:44.643259 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:53.138750 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:28:53.138716 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:53.364887 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:28:53.364856 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:29:03.520780 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:29:03.520750 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:13.823081 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:29:13.823050 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:23.364589 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:29:23.364552 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:29:28.693973 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:29:28.693945 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-69g25_da86486a-f4f9-4bc3-a5e0-be4073682c0c/payload-processing/0.log" Oct 30 20:29:28.916266 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:29:28.916240 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-69g25_da86486a-f4f9-4bc3-a5e0-be4073682c0c/payload-processing/0.log" Oct 30 20:29:44.982150 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:29:44.982119 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:53.364732 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:29:53.364695 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:29:53.458947 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:29:53.458919 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:14.151122 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:14.151089 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:23.364926 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:30:23.364893 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:30:45.319422 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:45.319356 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:48.314469 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:48.314444 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-69g25_da86486a-f4f9-4bc3-a5e0-be4073682c0c/payload-processing/0.log" Oct 30 20:30:48.527799 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:48.527773 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-69g25_da86486a-f4f9-4bc3-a5e0-be4073682c0c/payload-processing/0.log" Oct 30 20:30:53.364378 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:30:53.364343 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:30:53.800660 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:53.800586 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:58.894915 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:58.894881 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n79m4/must-gather-6whrn"] Oct 30 20:30:58.895694 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:58.895260 4433 cpu_manager.go:414] "containerMap: removing container" podUID="860c3b62-aa0e-466b-9991-029a5cf5a394" containerName="maas-api" Oct 30 20:30:58.895694 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:58.895271 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860c3b62-aa0e-466b-9991-029a5cf5a394" containerName="maas-api" Oct 30 20:30:58.895694 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:58.895294 4433 cpu_manager.go:414] "containerMap: removing container" podUID="860c3b62-aa0e-466b-9991-029a5cf5a394" containerName="maas-api" Oct 30 20:30:58.895694 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:58.895299 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860c3b62-aa0e-466b-9991-029a5cf5a394" containerName="maas-api" Oct 30 20:30:58.895694 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:58.895368 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="860c3b62-aa0e-466b-9991-029a5cf5a394" containerName="maas-api" Oct 30 20:30:58.895694 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:58.895547 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="860c3b62-aa0e-466b-9991-029a5cf5a394" containerName="maas-api" Oct 30 20:30:58.898713 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:58.898695 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n79m4/must-gather-6whrn" Oct 30 20:30:58.985367 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:58.985344 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:58.999672 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:58.999653 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n79m4\"/\"openshift-service-ca.crt\"" Oct 30 20:30:58.999672 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:58.999653 4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n79m4\"/\"kube-root-ca.crt\"" Oct 30 20:30:58.999831 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:58.999657 4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n79m4\"/\"default-dockercfg-jfgg8\"" Oct 30 20:30:59.001780 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:59.001755 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n79m4/must-gather-6whrn"] Oct 30 20:30:59.025563 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:59.025542 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/10ae249e-747e-4a26-a4b0-5afd65c3e460-must-gather-output\") pod \"must-gather-6whrn\" (UID: \"10ae249e-747e-4a26-a4b0-5afd65c3e460\") " pod="openshift-must-gather-n79m4/must-gather-6whrn" Oct 30 20:30:59.025667 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:59.025654 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q66kn\" (UniqueName: \"kubernetes.io/projected/10ae249e-747e-4a26-a4b0-5afd65c3e460-kube-api-access-q66kn\") pod \"must-gather-6whrn\" (UID: \"10ae249e-747e-4a26-a4b0-5afd65c3e460\") " pod="openshift-must-gather-n79m4/must-gather-6whrn" Oct 30 20:30:59.126561 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:59.126535 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q66kn\" (UniqueName: \"kubernetes.io/projected/10ae249e-747e-4a26-a4b0-5afd65c3e460-kube-api-access-q66kn\") pod \"must-gather-6whrn\" (UID: \"10ae249e-747e-4a26-a4b0-5afd65c3e460\") " pod="openshift-must-gather-n79m4/must-gather-6whrn" Oct 30 20:30:59.126651 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:59.126586 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/10ae249e-747e-4a26-a4b0-5afd65c3e460-must-gather-output\") pod \"must-gather-6whrn\" (UID: \"10ae249e-747e-4a26-a4b0-5afd65c3e460\") " pod="openshift-must-gather-n79m4/must-gather-6whrn" Oct 30 20:30:59.126876 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:59.126862 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/10ae249e-747e-4a26-a4b0-5afd65c3e460-must-gather-output\") pod \"must-gather-6whrn\" (UID: \"10ae249e-747e-4a26-a4b0-5afd65c3e460\") " pod="openshift-must-gather-n79m4/must-gather-6whrn" Oct 30 20:30:59.183331 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:59.183285 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q66kn\" (UniqueName: \"kubernetes.io/projected/10ae249e-747e-4a26-a4b0-5afd65c3e460-kube-api-access-q66kn\") pod \"must-gather-6whrn\" (UID: \"10ae249e-747e-4a26-a4b0-5afd65c3e460\") " pod="openshift-must-gather-n79m4/must-gather-6whrn" Oct 30 20:30:59.212104 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:59.212081 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n79m4/must-gather-6whrn" Oct 30 20:30:59.218223 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:59.218201 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:59.580350 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:59.580306 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n79m4/must-gather-6whrn"] Oct 30 20:30:59.582490 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:30:59.582470 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10ae249e_747e_4a26_a4b0_5afd65c3e460.slice/crio-2dc32456c1ea373999455162de5fb4039edfc41010722e31727cd4353471a242 WatchSource:0}: Error finding container 2dc32456c1ea373999455162de5fb4039edfc41010722e31727cd4353471a242: Status 404 returned error can't find the container with id 2dc32456c1ea373999455162de5fb4039edfc41010722e31727cd4353471a242 Oct 30 20:30:59.584244 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:30:59.584226 4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:31:00.472252 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:00.472222 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n79m4/must-gather-6whrn" event={"ID":"10ae249e-747e-4a26-a4b0-5afd65c3e460","Type":"ContainerStarted","Data":"2dc32456c1ea373999455162de5fb4039edfc41010722e31727cd4353471a242"} Oct 30 20:31:01.478248 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:01.478214 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n79m4/must-gather-6whrn" event={"ID":"10ae249e-747e-4a26-a4b0-5afd65c3e460","Type":"ContainerStarted","Data":"8caa754f3e7ce94b93fe26c65a64867140c95912acdd1868fb69865ff9844c94"} Oct 30 20:31:01.478248 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:01.478254 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n79m4/must-gather-6whrn" event={"ID":"10ae249e-747e-4a26-a4b0-5afd65c3e460","Type":"ContainerStarted","Data":"349dcf4fb610b47329d3eb2371fd50ffb17acba0e9369fba78268969881f88a9"} Oct 30 20:31:01.592485 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:01.592432 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-n79m4/must-gather-6whrn" podStartSLOduration=2.360561072 podStartE2EDuration="3.592419197s" totalImagesPullingTime="1.231858125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:01.589406841 +0000 UTC m=+4508.524194396" watchObservedRunningTime="2026-10-30 20:31:01.592419197 +0000 UTC m=+4508.527206767" Oct 30 20:31:07.584803 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:07.584767 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6vks7_ed1dc05f-2a13-4dd2-9d20-f5e96f17099f/global-pull-secret-syncer/0.log" Oct 30 20:31:10.697288 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:10.697261 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" Oct 30 20:31:12.891701 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:12.891669 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-5.us-west-1.compute.internal_d17eaea73d7bf32a8a9bb9e63451f939/haproxy/0.log" Oct 30 20:31:14.525915 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:14.525870 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:23.366119 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:31:23.366081 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:31:37.358055 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:37.357924 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z"] Oct 30 20:31:37.358697 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:37.358368 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" podUID="485179f9-ab6f-4ff6-8520-437297c69df5" containerName="istio-proxy" containerID="cri-o://9728b3783d1c2fa985397ea1d3f8964ed719d6967ce9a0bb0b724cee6c4758fe" gracePeriod=30 Oct 30 20:31:37.379630 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:37.373323 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk"] Oct 30 20:31:37.379630 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:37.373381 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x"] Oct 30 20:31:37.379630 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:37.373533 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" podUID="f7f198b8-0ce4-4ac4-9339-944e7f829bf1" containerName="istio-proxy" containerID="cri-o://a499b3bdacc88491e09d94c6eab2459bce23b99bfca2e320884e2605df336e56" gracePeriod=30 Oct 30 20:31:37.379630 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:37.373722 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerName="istio-proxy" containerID="cri-o://2f12f6b78f24f7ceb865e861616b9fd2c9d5413956fd84c34b4244ef50e3bd17" gracePeriod=30 Oct 30 20:31:37.405002 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:37.400173 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt"] Oct 30 20:31:37.405002 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:37.400407 4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" podUID="3485be44-cb71-42c4-ab8e-dfcd161784cd" containerName="istio-proxy" containerID="cri-o://680f2661ae4ffb440da502aab71c31417f02d5db2f57a1ccdea6941d32a663ae" gracePeriod=30 Oct 30 20:31:37.700068 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:37.697121 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fzmkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:31:37.700068 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:37.697180 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" podUID="3485be44-cb71-42c4-ab8e-dfcd161784cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:31:42.429597 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.429543 4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-pmt9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": read tcp 10.134.0.2:43258->10.134.0.86:15021: read: connection reset by peer" start-of-body= Oct 30 20:31:42.430209 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.429634 4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" podUID="f7f198b8-0ce4-4ac4-9339-944e7f829bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": read tcp 10.134.0.2:43258->10.134.0.86:15021: read: connection reset by peer" Oct 30 20:31:42.676594 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.676552 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:31:42.685756 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.685730 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 20:31:42.695433 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.695409 4433 generic.go:356] "Generic (PLEG): container finished" podID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerID="2f12f6b78f24f7ceb865e861616b9fd2c9d5413956fd84c34b4244ef50e3bd17" exitCode=0 Oct 30 20:31:42.695551 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.695488 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" event={"ID":"ef5e46b2-1008-4306-92fb-abfae43bcd79","Type":"ContainerDied","Data":"2f12f6b78f24f7ceb865e861616b9fd2c9d5413956fd84c34b4244ef50e3bd17"} Oct 30 20:31:42.697467 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.697114 4433 generic.go:356] "Generic (PLEG): container finished" podID="485179f9-ab6f-4ff6-8520-437297c69df5" containerID="9728b3783d1c2fa985397ea1d3f8964ed719d6967ce9a0bb0b724cee6c4758fe" exitCode=0 Oct 30 20:31:42.697467 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.697253 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" event={"ID":"485179f9-ab6f-4ff6-8520-437297c69df5","Type":"ContainerDied","Data":"9728b3783d1c2fa985397ea1d3f8964ed719d6967ce9a0bb0b724cee6c4758fe"} Oct 30 20:31:42.698613 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.698590 4433 generic.go:356] "Generic (PLEG): container finished" podID="f7f198b8-0ce4-4ac4-9339-944e7f829bf1" containerID="a499b3bdacc88491e09d94c6eab2459bce23b99bfca2e320884e2605df336e56" exitCode=0 Oct 30 20:31:42.698714 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.698635 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" Oct 30 20:31:42.698714 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.698656 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" event={"ID":"f7f198b8-0ce4-4ac4-9339-944e7f829bf1","Type":"ContainerDied","Data":"a499b3bdacc88491e09d94c6eab2459bce23b99bfca2e320884e2605df336e56"} Oct 30 20:31:42.698714 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.698708 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x" event={"ID":"f7f198b8-0ce4-4ac4-9339-944e7f829bf1","Type":"ContainerDied","Data":"0ca00d30d29ea9b44992f25603c3aa242c17fb4c4fe873612c95b371b858fc51"} Oct 30 20:31:42.698884 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.698739 4433 scope.go:122] "RemoveContainer" containerID="a499b3bdacc88491e09d94c6eab2459bce23b99bfca2e320884e2605df336e56" Oct 30 20:31:42.700011 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.699992 4433 generic.go:356] "Generic (PLEG): container finished" podID="3485be44-cb71-42c4-ab8e-dfcd161784cd" containerID="680f2661ae4ffb440da502aab71c31417f02d5db2f57a1ccdea6941d32a663ae" exitCode=0 Oct 30 20:31:42.700108 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.700023 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" event={"ID":"3485be44-cb71-42c4-ab8e-dfcd161784cd","Type":"ContainerDied","Data":"680f2661ae4ffb440da502aab71c31417f02d5db2f57a1ccdea6941d32a663ae"} Oct 30 20:31:42.700108 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.700030 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" Oct 30 20:31:42.700108 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.700059 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt" event={"ID":"3485be44-cb71-42c4-ab8e-dfcd161784cd","Type":"ContainerDied","Data":"dfe105ffc3306c0975298fe47cc1e6230a7f494feedb00c3c8d0a59621cc25f7"} Oct 30 20:31:42.709433 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.709410 4433 scope.go:122] "RemoveContainer" containerID="a499b3bdacc88491e09d94c6eab2459bce23b99bfca2e320884e2605df336e56" Oct 30 20:31:42.709742 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:31:42.709719 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a499b3bdacc88491e09d94c6eab2459bce23b99bfca2e320884e2605df336e56\": container with ID starting with a499b3bdacc88491e09d94c6eab2459bce23b99bfca2e320884e2605df336e56 not found: ID does not exist" containerID="a499b3bdacc88491e09d94c6eab2459bce23b99bfca2e320884e2605df336e56" Oct 30 20:31:42.709837 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.709743 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a499b3bdacc88491e09d94c6eab2459bce23b99bfca2e320884e2605df336e56"} err="failed to get container status \"a499b3bdacc88491e09d94c6eab2459bce23b99bfca2e320884e2605df336e56\": rpc error: code = NotFound desc = could not find container \"a499b3bdacc88491e09d94c6eab2459bce23b99bfca2e320884e2605df336e56\": container with ID starting with a499b3bdacc88491e09d94c6eab2459bce23b99bfca2e320884e2605df336e56 not found: ID does not exist" Oct 30 20:31:42.709837 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.709759 4433 scope.go:122] "RemoveContainer" containerID="680f2661ae4ffb440da502aab71c31417f02d5db2f57a1ccdea6941d32a663ae" Oct 30 20:31:42.720736 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.719899 4433 scope.go:122] "RemoveContainer" containerID="680f2661ae4ffb440da502aab71c31417f02d5db2f57a1ccdea6941d32a663ae" Oct 30 20:31:42.720736 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:31:42.720223 4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680f2661ae4ffb440da502aab71c31417f02d5db2f57a1ccdea6941d32a663ae\": container with ID starting with 680f2661ae4ffb440da502aab71c31417f02d5db2f57a1ccdea6941d32a663ae not found: ID does not exist" containerID="680f2661ae4ffb440da502aab71c31417f02d5db2f57a1ccdea6941d32a663ae" Oct 30 20:31:42.720736 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.720246 4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680f2661ae4ffb440da502aab71c31417f02d5db2f57a1ccdea6941d32a663ae"} err="failed to get container status \"680f2661ae4ffb440da502aab71c31417f02d5db2f57a1ccdea6941d32a663ae\": rpc error: code = NotFound desc = could not find container \"680f2661ae4ffb440da502aab71c31417f02d5db2f57a1ccdea6941d32a663ae\": container with ID starting with 680f2661ae4ffb440da502aab71c31417f02d5db2f57a1ccdea6941d32a663ae not found: ID does not exist" Oct 30 20:31:42.773250 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773224 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-podinfo\") pod \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " Oct 30 20:31:42.773376 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773261 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istiod-ca-cert\") pod \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " Oct 30 20:31:42.773376 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773283 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-workload-certs\") pod \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " Oct 30 20:31:42.773376 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773321 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-workload-certs\") pod \"3485be44-cb71-42c4-ab8e-dfcd161784cd\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " Oct 30 20:31:42.773376 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773370 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3485be44-cb71-42c4-ab8e-dfcd161784cd-kube-api-access-bdd6z\" (UniqueName: \"kubernetes.io/projected/3485be44-cb71-42c4-ab8e-dfcd161784cd-kube-api-access-bdd6z\") pod \"3485be44-cb71-42c4-ab8e-dfcd161784cd\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " Oct 30 20:31:42.773560 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773404 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-data\") pod \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " Oct 30 20:31:42.773560 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773438 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-data\") pod \"3485be44-cb71-42c4-ab8e-dfcd161784cd\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " Oct 30 20:31:42.773560 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773482 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-workload-socket\") pod \"3485be44-cb71-42c4-ab8e-dfcd161784cd\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " Oct 30 20:31:42.773560 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773527 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-credential-socket\") pod \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " Oct 30 20:31:42.773773 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773587 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-envoy\") pod \"3485be44-cb71-42c4-ab8e-dfcd161784cd\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " Oct 30 20:31:42.773773 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773615 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-workload-socket\") pod \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " Oct 30 20:31:42.773773 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773615 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-workload-certs" pod "3485be44-cb71-42c4-ab8e-dfcd161784cd" (UID: "3485be44-cb71-42c4-ab8e-dfcd161784cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.773773 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773637 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-data" pod "f7f198b8-0ce4-4ac4-9339-944e7f829bf1" (UID: "f7f198b8-0ce4-4ac4-9339-944e7f829bf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.773773 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773622 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-workload-certs" pod "f7f198b8-0ce4-4ac4-9339-944e7f829bf1" (UID: "f7f198b8-0ce4-4ac4-9339-944e7f829bf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.773773 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773661 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-kube-api-access-ww5n7\" (UniqueName: \"kubernetes.io/projected/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-kube-api-access-ww5n7\") pod \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " Oct 30 20:31:42.773773 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773718 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3485be44-cb71-42c4-ab8e-dfcd161784cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3485be44-cb71-42c4-ab8e-dfcd161784cd-istiod-ca-cert\") pod \"3485be44-cb71-42c4-ab8e-dfcd161784cd\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " Oct 30 20:31:42.773773 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773760 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-envoy\") pod \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " Oct 30 20:31:42.774249 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773747 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-data" pod "3485be44-cb71-42c4-ab8e-dfcd161784cd" (UID: "3485be44-cb71-42c4-ab8e-dfcd161784cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.774249 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773796 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-credential-socket\") pod \"3485be44-cb71-42c4-ab8e-dfcd161784cd\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " Oct 30 20:31:42.774249 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773798 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-credential-socket" pod "f7f198b8-0ce4-4ac4-9339-944e7f829bf1" (UID: "f7f198b8-0ce4-4ac4-9339-944e7f829bf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.774249 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773826 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-workload-socket" pod "3485be44-cb71-42c4-ab8e-dfcd161784cd" (UID: "3485be44-cb71-42c4-ab8e-dfcd161784cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.774249 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773840 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-token\" (UniqueName: \"kubernetes.io/projected/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-token\") pod \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\" (UID: \"f7f198b8-0ce4-4ac4-9339-944e7f829bf1\") " Oct 30 20:31:42.774249 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773889 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-token\" (UniqueName: \"kubernetes.io/projected/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-token\") pod \"3485be44-cb71-42c4-ab8e-dfcd161784cd\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " Oct 30 20:31:42.774249 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.773924 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-podinfo\") pod \"3485be44-cb71-42c4-ab8e-dfcd161784cd\" (UID: \"3485be44-cb71-42c4-ab8e-dfcd161784cd\") " Oct 30 20:31:42.774249 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.774116 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3485be44-cb71-42c4-ab8e-dfcd161784cd-istiod-ca-cert" pod "3485be44-cb71-42c4-ab8e-dfcd161784cd" (UID: "3485be44-cb71-42c4-ab8e-dfcd161784cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:42.774683 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.774269 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-workload-socket" pod "f7f198b8-0ce4-4ac4-9339-944e7f829bf1" (UID: "f7f198b8-0ce4-4ac4-9339-944e7f829bf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.774683 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.774375 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.774683 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.774394 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.774683 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.774410 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.774683 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.774424 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3485be44-cb71-42c4-ab8e-dfcd161784cd-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.774683 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.774440 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.774683 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.774454 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.774683 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.774470 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.774683 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.774485 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.774683 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.774502 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-credential-socket" pod "3485be44-cb71-42c4-ab8e-dfcd161784cd" (UID: "3485be44-cb71-42c4-ab8e-dfcd161784cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.775244 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.774734 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istiod-ca-cert" pod "f7f198b8-0ce4-4ac4-9339-944e7f829bf1" (UID: "f7f198b8-0ce4-4ac4-9339-944e7f829bf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:42.776245 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.776204 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3485be44-cb71-42c4-ab8e-dfcd161784cd-kube-api-access-bdd6z" pod "3485be44-cb71-42c4-ab8e-dfcd161784cd" (UID: "3485be44-cb71-42c4-ab8e-dfcd161784cd"). InnerVolumeSpecName "kube-api-access-bdd6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:42.776432 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.776410 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-envoy" pod "f7f198b8-0ce4-4ac4-9339-944e7f829bf1" (UID: "f7f198b8-0ce4-4ac4-9339-944e7f829bf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.776614 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.776592 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-envoy" pod "3485be44-cb71-42c4-ab8e-dfcd161784cd" (UID: "3485be44-cb71-42c4-ab8e-dfcd161784cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.776614 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.776604 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-podinfo" pod "f7f198b8-0ce4-4ac4-9339-944e7f829bf1" (UID: "f7f198b8-0ce4-4ac4-9339-944e7f829bf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:31:42.776749 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.776591 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-kube-api-access-ww5n7" pod "f7f198b8-0ce4-4ac4-9339-944e7f829bf1" (UID: "f7f198b8-0ce4-4ac4-9339-944e7f829bf1"). InnerVolumeSpecName "kube-api-access-ww5n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:42.776749 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.776724 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-token" pod "3485be44-cb71-42c4-ab8e-dfcd161784cd" (UID: "3485be44-cb71-42c4-ab8e-dfcd161784cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:42.776948 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.776929 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-podinfo" pod "3485be44-cb71-42c4-ab8e-dfcd161784cd" (UID: "3485be44-cb71-42c4-ab8e-dfcd161784cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:31:42.777666 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.777642 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-token" pod "f7f198b8-0ce4-4ac4-9339-944e7f829bf1" (UID: "f7f198b8-0ce4-4ac4-9339-944e7f829bf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:42.875475 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.875432 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww5n7\" (UniqueName: \"kubernetes.io/projected/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-kube-api-access-ww5n7\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.875475 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.875473 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.875475 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.875484 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.875475 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.875493 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.875731 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.875501 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.875731 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.875510 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.875731 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.875518 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.875731 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.875528 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f198b8-0ce4-4ac4-9339-944e7f829bf1-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.875731 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.875536 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdd6z\" (UniqueName: \"kubernetes.io/projected/3485be44-cb71-42c4-ab8e-dfcd161784cd-kube-api-access-bdd6z\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:42.875731 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:42.875545 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3485be44-cb71-42c4-ab8e-dfcd161784cd-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.088015 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.085469 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:31:43.090544 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.090515 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 20:31:43.130705 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.130663 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x"] Oct 30 20:31:43.133492 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.133459 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-pmt9x"] Oct 30 20:31:43.177812 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.177731 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-workload-certs\") pod \"ef5e46b2-1008-4306-92fb-abfae43bcd79\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " Oct 30 20:31:43.177812 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.177773 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-credential-socket\") pod \"485179f9-ab6f-4ff6-8520-437297c69df5\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " Oct 30 20:31:43.177812 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.177798 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-workload-socket\") pod \"485179f9-ab6f-4ff6-8520-437297c69df5\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " Oct 30 20:31:43.178069 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.177835 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-envoy\") pod \"ef5e46b2-1008-4306-92fb-abfae43bcd79\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " Oct 30 20:31:43.178069 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.177867 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-credential-socket\") pod \"ef5e46b2-1008-4306-92fb-abfae43bcd79\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " Oct 30 20:31:43.178069 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.177944 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-data\") pod \"ef5e46b2-1008-4306-92fb-abfae43bcd79\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " Oct 30 20:31:43.178069 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.177970 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/485179f9-ab6f-4ff6-8520-437297c69df5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/485179f9-ab6f-4ff6-8520-437297c69df5-istiod-ca-cert\") pod \"485179f9-ab6f-4ff6-8520-437297c69df5\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " Oct 30 20:31:43.178069 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178041 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/485179f9-ab6f-4ff6-8520-437297c69df5-kube-api-access-bd6qn\" (UniqueName: \"kubernetes.io/projected/485179f9-ab6f-4ff6-8520-437297c69df5-kube-api-access-bd6qn\") pod \"485179f9-ab6f-4ff6-8520-437297c69df5\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " Oct 30 20:31:43.178069 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178054 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-credential-socket" pod "485179f9-ab6f-4ff6-8520-437297c69df5" (UID: "485179f9-ab6f-4ff6-8520-437297c69df5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:43.178289 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178076 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/485179f9-ab6f-4ff6-8520-437297c69df5-istio-token\" (UniqueName: \"kubernetes.io/projected/485179f9-ab6f-4ff6-8520-437297c69df5-istio-token\") pod \"485179f9-ab6f-4ff6-8520-437297c69df5\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " Oct 30 20:31:43.178289 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178095 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-workload-certs" pod "ef5e46b2-1008-4306-92fb-abfae43bcd79" (UID: "ef5e46b2-1008-4306-92fb-abfae43bcd79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:43.178289 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178119 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-podinfo\") pod \"ef5e46b2-1008-4306-92fb-abfae43bcd79\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " Oct 30 20:31:43.178289 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178146 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-istio-envoy\") pod \"485179f9-ab6f-4ff6-8520-437297c69df5\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " Oct 30 20:31:43.178439 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178421 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-data" pod "ef5e46b2-1008-4306-92fb-abfae43bcd79" (UID: "ef5e46b2-1008-4306-92fb-abfae43bcd79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:43.178473 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178433 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/485179f9-ab6f-4ff6-8520-437297c69df5-istiod-ca-cert" pod "485179f9-ab6f-4ff6-8520-437297c69df5" (UID: "485179f9-ab6f-4ff6-8520-437297c69df5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:43.178507 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178471 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-credential-socket" pod "ef5e46b2-1008-4306-92fb-abfae43bcd79" (UID: "ef5e46b2-1008-4306-92fb-abfae43bcd79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:43.178590 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178570 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-istio-data\") pod \"485179f9-ab6f-4ff6-8520-437297c69df5\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " Oct 30 20:31:43.178715 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178655 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-workload-socket\") pod \"ef5e46b2-1008-4306-92fb-abfae43bcd79\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " Oct 30 20:31:43.178715 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178688 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef5e46b2-1008-4306-92fb-abfae43bcd79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef5e46b2-1008-4306-92fb-abfae43bcd79-istiod-ca-cert\") pod \"ef5e46b2-1008-4306-92fb-abfae43bcd79\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " Oct 30 20:31:43.178831 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178723 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-workload-certs\") pod \"485179f9-ab6f-4ff6-8520-437297c69df5\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " Oct 30 20:31:43.178831 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178759 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-token\" (UniqueName: \"kubernetes.io/projected/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-token\") pod \"ef5e46b2-1008-4306-92fb-abfae43bcd79\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " Oct 30 20:31:43.178831 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178786 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/485179f9-ab6f-4ff6-8520-437297c69df5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/485179f9-ab6f-4ff6-8520-437297c69df5-istio-podinfo\") pod \"485179f9-ab6f-4ff6-8520-437297c69df5\" (UID: \"485179f9-ab6f-4ff6-8520-437297c69df5\") " Oct 30 20:31:43.178831 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178814 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-istio-data" pod "485179f9-ab6f-4ff6-8520-437297c69df5" (UID: "485179f9-ab6f-4ff6-8520-437297c69df5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:43.179072 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.178819 4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5e46b2-1008-4306-92fb-abfae43bcd79-kube-api-access-bt2fd\" (UniqueName: \"kubernetes.io/projected/ef5e46b2-1008-4306-92fb-abfae43bcd79-kube-api-access-bt2fd\") pod \"ef5e46b2-1008-4306-92fb-abfae43bcd79\" (UID: \"ef5e46b2-1008-4306-92fb-abfae43bcd79\") " Oct 30 20:31:43.179132 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.179101 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-workload-socket" pod "485179f9-ab6f-4ff6-8520-437297c69df5" (UID: "485179f9-ab6f-4ff6-8520-437297c69df5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:43.179354 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.179330 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-workload-certs" pod "485179f9-ab6f-4ff6-8520-437297c69df5" (UID: "485179f9-ab6f-4ff6-8520-437297c69df5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:43.179451 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.179355 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.179451 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.179380 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.179451 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.179396 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/485179f9-ab6f-4ff6-8520-437297c69df5-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.179451 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.179412 4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-istio-data\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.179451 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.179428 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.179451 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.179443 4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-credential-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.179775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.179457 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.179775 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.179524 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-workload-socket" pod "ef5e46b2-1008-4306-92fb-abfae43bcd79" (UID: "ef5e46b2-1008-4306-92fb-abfae43bcd79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:43.179893 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.179770 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5e46b2-1008-4306-92fb-abfae43bcd79-istiod-ca-cert" pod "ef5e46b2-1008-4306-92fb-abfae43bcd79" (UID: "ef5e46b2-1008-4306-92fb-abfae43bcd79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:43.181437 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.181356 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5e46b2-1008-4306-92fb-abfae43bcd79-kube-api-access-bt2fd" pod "ef5e46b2-1008-4306-92fb-abfae43bcd79" (UID: "ef5e46b2-1008-4306-92fb-abfae43bcd79"). InnerVolumeSpecName "kube-api-access-bt2fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:43.181437 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.181377 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485179f9-ab6f-4ff6-8520-437297c69df5-kube-api-access-bd6qn" pod "485179f9-ab6f-4ff6-8520-437297c69df5" (UID: "485179f9-ab6f-4ff6-8520-437297c69df5"). InnerVolumeSpecName "kube-api-access-bd6qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:43.181437 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.181393 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485179f9-ab6f-4ff6-8520-437297c69df5-istio-token" pod "485179f9-ab6f-4ff6-8520-437297c69df5" (UID: "485179f9-ab6f-4ff6-8520-437297c69df5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:43.181708 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.181689 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-podinfo" pod "ef5e46b2-1008-4306-92fb-abfae43bcd79" (UID: "ef5e46b2-1008-4306-92fb-abfae43bcd79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:31:43.181922 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.181895 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-token" pod "ef5e46b2-1008-4306-92fb-abfae43bcd79" (UID: "ef5e46b2-1008-4306-92fb-abfae43bcd79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:43.182871 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.182835 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-istio-envoy" pod "485179f9-ab6f-4ff6-8520-437297c69df5" (UID: "485179f9-ab6f-4ff6-8520-437297c69df5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:43.183199 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.183178 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-envoy" pod "ef5e46b2-1008-4306-92fb-abfae43bcd79" (UID: "ef5e46b2-1008-4306-92fb-abfae43bcd79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:43.183271 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.183219 4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/485179f9-ab6f-4ff6-8520-437297c69df5-istio-podinfo" pod "485179f9-ab6f-4ff6-8520-437297c69df5" (UID: "485179f9-ab6f-4ff6-8520-437297c69df5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:31:43.213268 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.213234 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f198b8-0ce4-4ac4-9339-944e7f829bf1" path="/var/lib/kubelet/pods/f7f198b8-0ce4-4ac4-9339-944e7f829bf1/volumes" Oct 30 20:31:43.280352 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.280313 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd6qn\" (UniqueName: \"kubernetes.io/projected/485179f9-ab6f-4ff6-8520-437297c69df5-kube-api-access-bd6qn\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.280352 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.280355 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/485179f9-ab6f-4ff6-8520-437297c69df5-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.280546 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.280371 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.280546 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.280387 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.280546 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.280403 4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-workload-socket\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.280546 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.280417 4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef5e46b2-1008-4306-92fb-abfae43bcd79-istiod-ca-cert\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.280546 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.280433 4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/485179f9-ab6f-4ff6-8520-437297c69df5-workload-certs\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.280546 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.280446 4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-token\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.280546 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.280459 4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/485179f9-ab6f-4ff6-8520-437297c69df5-istio-podinfo\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.280546 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.280473 4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt2fd\" (UniqueName: \"kubernetes.io/projected/ef5e46b2-1008-4306-92fb-abfae43bcd79-kube-api-access-bt2fd\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.280546 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.280487 4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef5e46b2-1008-4306-92fb-abfae43bcd79-istio-envoy\") on node \"ip-10-0-141-5.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:31:43.360214 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.360179 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt"] Oct 30 20:31:43.364928 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.364894 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fzmkt"] Oct 30 20:31:43.707077 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.707042 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" Oct 30 20:31:43.707555 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.707053 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk" event={"ID":"ef5e46b2-1008-4306-92fb-abfae43bcd79","Type":"ContainerDied","Data":"b7e17b5d3034477ff779ed8580556defde0f06048941d92d2849866aaf936434"} Oct 30 20:31:43.707555 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.707172 4433 scope.go:122] "RemoveContainer" containerID="2f12f6b78f24f7ceb865e861616b9fd2c9d5413956fd84c34b4244ef50e3bd17" Oct 30 20:31:43.708435 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.708343 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" event={"ID":"485179f9-ab6f-4ff6-8520-437297c69df5","Type":"ContainerDied","Data":"52cf0c5637d37eb75cda2ece668c56565b78cf4da603e58c0b7ae1a504b69fa4"} Oct 30 20:31:43.708435 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.708376 4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z" Oct 30 20:31:43.716941 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.716925 4433 scope.go:122] "RemoveContainer" containerID="9728b3783d1c2fa985397ea1d3f8964ed719d6967ce9a0bb0b724cee6c4758fe" Oct 30 20:31:43.869315 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.869280 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z"] Oct 30 20:31:43.872517 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.872489 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ld68z"] Oct 30 20:31:43.980685 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.980605 4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk"] Oct 30 20:31:43.985688 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:43.985655 4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r7gtk"] Oct 30 20:31:45.212479 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.212444 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3485be44-cb71-42c4-ab8e-dfcd161784cd" path="/var/lib/kubelet/pods/3485be44-cb71-42c4-ab8e-dfcd161784cd/volumes" Oct 30 20:31:45.212970 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.212954 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="485179f9-ab6f-4ff6-8520-437297c69df5" path="/var/lib/kubelet/pods/485179f9-ab6f-4ff6-8520-437297c69df5/volumes" Oct 30 20:31:45.213442 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.213426 4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" path="/var/lib/kubelet/pods/ef5e46b2-1008-4306-92fb-abfae43bcd79/volumes" Oct 30 20:31:45.656348 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.656313 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:45.762779 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.762742 4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw"] Oct 30 20:31:45.763191 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.763175 4433 cpu_manager.go:414] "containerMap: removing container" podUID="f7f198b8-0ce4-4ac4-9339-944e7f829bf1" containerName="istio-proxy" Oct 30 20:31:45.763253 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.763192 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f198b8-0ce4-4ac4-9339-944e7f829bf1" containerName="istio-proxy" Oct 30 20:31:45.763253 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.763204 4433 cpu_manager.go:414] "containerMap: removing container" podUID="485179f9-ab6f-4ff6-8520-437297c69df5" containerName="istio-proxy" Oct 30 20:31:45.763253 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.763210 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="485179f9-ab6f-4ff6-8520-437297c69df5" containerName="istio-proxy" Oct 30 20:31:45.763253 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.763229 4433 cpu_manager.go:414] "containerMap: removing container" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerName="istio-proxy" Oct 30 20:31:45.763253 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.763234 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerName="istio-proxy" Oct 30 20:31:45.763253 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.763248 4433 cpu_manager.go:414] "containerMap: removing container" podUID="3485be44-cb71-42c4-ab8e-dfcd161784cd" containerName="istio-proxy" Oct 30 20:31:45.763253 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.763253 4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3485be44-cb71-42c4-ab8e-dfcd161784cd" containerName="istio-proxy" Oct 30 20:31:45.763620 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.763309 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5e46b2-1008-4306-92fb-abfae43bcd79" containerName="istio-proxy" Oct 30 20:31:45.763620 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.763317 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3485be44-cb71-42c4-ab8e-dfcd161784cd" containerName="istio-proxy" Oct 30 20:31:45.763620 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.763328 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f198b8-0ce4-4ac4-9339-944e7f829bf1" containerName="istio-proxy" Oct 30 20:31:45.763620 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.763334 4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="485179f9-ab6f-4ff6-8520-437297c69df5" containerName="istio-proxy" Oct 30 20:31:45.768914 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.768887 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:45.823714 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.823677 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw"] Oct 30 20:31:45.903376 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.903345 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f968facb-2d9c-47a9-9169-8942eaa64d14-lib-modules\") pod \"perf-node-gather-daemonset-zhkvw\" (UID: \"f968facb-2d9c-47a9-9169-8942eaa64d14\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:45.903556 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.903434 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f968facb-2d9c-47a9-9169-8942eaa64d14-proc\") pod \"perf-node-gather-daemonset-zhkvw\" (UID: \"f968facb-2d9c-47a9-9169-8942eaa64d14\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:45.903556 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.903486 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f968facb-2d9c-47a9-9169-8942eaa64d14-podres\") pod \"perf-node-gather-daemonset-zhkvw\" (UID: \"f968facb-2d9c-47a9-9169-8942eaa64d14\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:45.903556 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.903547 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8td\" (UniqueName: \"kubernetes.io/projected/f968facb-2d9c-47a9-9169-8942eaa64d14-kube-api-access-bw8td\") pod \"perf-node-gather-daemonset-zhkvw\" (UID: \"f968facb-2d9c-47a9-9169-8942eaa64d14\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:45.903688 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:45.903621 4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f968facb-2d9c-47a9-9169-8942eaa64d14-sys\") pod \"perf-node-gather-daemonset-zhkvw\" (UID: \"f968facb-2d9c-47a9-9169-8942eaa64d14\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:46.004606 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:46.004511 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f968facb-2d9c-47a9-9169-8942eaa64d14-lib-modules\") pod \"perf-node-gather-daemonset-zhkvw\" (UID: \"f968facb-2d9c-47a9-9169-8942eaa64d14\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:46.004785 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:46.004608 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f968facb-2d9c-47a9-9169-8942eaa64d14-proc\") pod \"perf-node-gather-daemonset-zhkvw\" (UID: \"f968facb-2d9c-47a9-9169-8942eaa64d14\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:46.004785 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:46.004637 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f968facb-2d9c-47a9-9169-8942eaa64d14-podres\") pod \"perf-node-gather-daemonset-zhkvw\" (UID: \"f968facb-2d9c-47a9-9169-8942eaa64d14\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:46.004785 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:46.004661 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f968facb-2d9c-47a9-9169-8942eaa64d14-lib-modules\") pod \"perf-node-gather-daemonset-zhkvw\" (UID: \"f968facb-2d9c-47a9-9169-8942eaa64d14\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:46.004785 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:46.004690 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8td\" (UniqueName: \"kubernetes.io/projected/f968facb-2d9c-47a9-9169-8942eaa64d14-kube-api-access-bw8td\") pod \"perf-node-gather-daemonset-zhkvw\" (UID: \"f968facb-2d9c-47a9-9169-8942eaa64d14\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:46.004785 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:46.004720 4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f968facb-2d9c-47a9-9169-8942eaa64d14-sys\") pod \"perf-node-gather-daemonset-zhkvw\" (UID: \"f968facb-2d9c-47a9-9169-8942eaa64d14\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:46.004785 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:46.004729 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f968facb-2d9c-47a9-9169-8942eaa64d14-podres\") pod \"perf-node-gather-daemonset-zhkvw\" (UID: \"f968facb-2d9c-47a9-9169-8942eaa64d14\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:46.004785 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:46.004722 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f968facb-2d9c-47a9-9169-8942eaa64d14-proc\") pod \"perf-node-gather-daemonset-zhkvw\" (UID: \"f968facb-2d9c-47a9-9169-8942eaa64d14\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:46.004785 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:46.004783 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f968facb-2d9c-47a9-9169-8942eaa64d14-sys\") pod \"perf-node-gather-daemonset-zhkvw\" (UID: \"f968facb-2d9c-47a9-9169-8942eaa64d14\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:46.060192 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:46.060160 4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8td\" (UniqueName: \"kubernetes.io/projected/f968facb-2d9c-47a9-9169-8942eaa64d14-kube-api-access-bw8td\") pod \"perf-node-gather-daemonset-zhkvw\" (UID: \"f968facb-2d9c-47a9-9169-8942eaa64d14\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:46.082387 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:46.082353 4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:46.453695 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:46.453608 4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw"] Oct 30 20:31:46.456634 ip-10-0-141-5 kubenswrapper[4433]: W1030 20:31:46.456605 4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf968facb_2d9c_47a9_9169_8942eaa64d14.slice/crio-4c27989006e463ec28643fbe124524693fac7cdb1d6d4211e71b4030d636e6e9 WatchSource:0}: Error finding container 4c27989006e463ec28643fbe124524693fac7cdb1d6d4211e71b4030d636e6e9: Status 404 returned error can't find the container with id 4c27989006e463ec28643fbe124524693fac7cdb1d6d4211e71b4030d636e6e9 Oct 30 20:31:46.729306 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:46.729220 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" event={"ID":"f968facb-2d9c-47a9-9169-8942eaa64d14","Type":"ContainerStarted","Data":"4c20e2f53df6e3a935ac16e5f3d67df1cddc1671419197051bec942192d137a6"} Oct 30 20:31:46.729306 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:46.729270 4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" event={"ID":"f968facb-2d9c-47a9-9169-8942eaa64d14","Type":"ContainerStarted","Data":"4c27989006e463ec28643fbe124524693fac7cdb1d6d4211e71b4030d636e6e9"} Oct 30 20:31:46.729507 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:46.729376 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:46.842664 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:46.842599 4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" podStartSLOduration=1.8425839819999998 podStartE2EDuration="1.842583982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:46.841159969 +0000 UTC m=+4553.775947495" watchObservedRunningTime="2026-10-30 20:31:46.842583982 +0000 UTC m=+4553.777371539" Oct 30 20:31:52.744883 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:52.744851 4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-zhkvw" Oct 30 20:31:53.367298 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:31:53.367259 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:31:54.066354 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:31:54.066317 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:14.300824 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:32:14.300792 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5xf482_40b46873-539f-42cb-856a-f28941276775/istio-proxy/0.log" Oct 30 20:32:17.610812 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:32:17.610777 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-t2t65_f8a5fdf4-c366-4af9-b351-15a6617824d7/payload-pre-processing/0.log" Oct 30 20:32:17.822604 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:32:17.822560 4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-69g25_da86486a-f4f9-4bc3-a5e0-be4073682c0c/payload-processing/0.log" Oct 30 20:32:23.367764 ip-10-0-141-5 kubenswrapper[4433]: E1030 20:32:23.367726 4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjh7f" containerName="extractor" Oct 30 20:32:24.838180 ip-10-0-141-5 kubenswrapper[4433]: I1030 20:32:24.838146 4433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4mc_ad299622-53d4-4610-b03a-3c586c0fff56/konnectivity-agent/0.log" limit=1024